Oct 30 19:41:00.628066 ip-10-0-133-27 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:41:01.032143 ip-10-0-133-27 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:01.032143 ip-10-0-133-27 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:41:01.032143 ip-10-0-133-27 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:01.032143 ip-10-0-133-27 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:01.038709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038594    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.038709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038708    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.038709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038712    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038717    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038720    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038723    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038725    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038728    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038731    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038734    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038736    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038739    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038742    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038745    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038747    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038750    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038753    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038756    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038759    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038761    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038764    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038779    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.038818 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038783    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038786    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038789    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038792    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038795    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038797    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038805    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038808    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038811    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038814    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038817    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038819    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038822    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038825    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038829    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038832    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038835    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038837    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038840    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038842    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.039277 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038845    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038847    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038850    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038852    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038855    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038858    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038860    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038863    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038865    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038868    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038870    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038873    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038877    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038882    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038886    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038890    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038893    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038895    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038898    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.039746 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038901    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038905    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038910    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038914    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038918    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038921    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038924    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038926    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038929    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038932    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038935    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038937    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038940    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038942    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038945    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038947    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038950    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038953    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038955    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.040212 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038958    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038960    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038963    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038965    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038968    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038971    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038974    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038977    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038979    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038982    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038985    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038987    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038990    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038992    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.038997    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039000    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039002    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039005    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039007    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039010    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.040655 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039012    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039015    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039017    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039019    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039022    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039024    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039834    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039842    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039845    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039848    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039850    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039853    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039856    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039858    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039861    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039864    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039867    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039869    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039873    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.041264 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039876    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039878    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039881    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039883    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039886    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039889    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039891    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039894    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039897    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039900    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039903    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039905    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039908    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039910    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039913    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039916    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039918    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039921    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039923    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039926    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.041959 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039928    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039931    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039934    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039936    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039939    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039941    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039944    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039948    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039952    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039955    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039958    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039960    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039963    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039966    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039969    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039971    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039974    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039977    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039980    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039983    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.042480 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039986    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039989    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039992    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039994    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039997    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.039999    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040002    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040004    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040007    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040010    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040013    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040015    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040018    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040021    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040023    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040026    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040028    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040031    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040037    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.042989 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040039    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040042    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040045    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040047    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040050    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040052    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040055    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040057    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040060    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040062    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040065    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040068    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040070    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040073    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040075    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040078    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040081    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040084    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040087    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040089    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.043442 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040092    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040094    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040097    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040099    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040103    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040107    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040111    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040113    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040116    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040118    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040121    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040126    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040129    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040131    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040623    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040628    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040631    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040633    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040636    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040639    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.043942 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040642    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040644    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040647    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040650    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040652    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040655    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040658    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.040700    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041308    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041319    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041326    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041331    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041335    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041340    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041345    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041350    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041355    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041360    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041364    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041375    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.044507 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041382    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041388    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041393    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041399    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041404    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041408    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041413    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041417    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041421    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041425    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041429    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041438    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041442    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041446    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041453    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041458    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041462    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041466    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041471    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.045537 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041475    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041484    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041490    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041494    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041503    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041508    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041513    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041517    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041521    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041525    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041529    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041533    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041538    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041542    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041546    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041550    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041555    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041564    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041568    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.046024 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041573    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041577    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041582    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041586    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041590    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041594    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041598    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041602    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041610    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041615    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041624    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041628    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041633    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041637    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041641    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041646    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041651    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041655    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041659    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.046467 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041664    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041668    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041672    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041680    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041684    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041689    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041693    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041697    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041701    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041705    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041709    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041713    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041717    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041721    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041726    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041730    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041738    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041742    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041746    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041750    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.046981 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041754    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041759    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.041763    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043076    2568 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043096    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043105    2568 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043110    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043115    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043119    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043124    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043129    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043132    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043136    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043139    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043143    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043146    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043150    2568 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043153    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043156    2568 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043159    2568 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043162    2568 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043168    2568 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043172    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043176    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:41:01.047446 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043179    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043183    2568 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043187    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043191    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043195    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043198    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043202    2568 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043205    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043208    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043211    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043214    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043220    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043224    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043227    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043230    2568 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043233    2568 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043236    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043241    2568 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043244    2568 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043247    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043251    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043254    2568 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043259    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043262    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043266    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:41:01.048068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043269    2568 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043272    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043275    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043278    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043281    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043289    2568 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043293    2568 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043296    2568 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043304    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043307    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043311    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043314    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043317    2568 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043321    2568 flags.go:64] FLAG: --help="false"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043324    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043327    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043330    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043333    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043337    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043341    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043344    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043348    2568 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043351    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:41:01.048685 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043354    2568 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043357    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043360    2568 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043363    2568 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043367    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043370    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043373    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043375    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043378    2568 flags.go:64] FLAG: --lock-file=""
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043381    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043384    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043387    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043393    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043396    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043399    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043402    2568 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043405    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043408    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043411    2568 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043414    2568 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043420    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043423    2568 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043428    2568 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043431    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043434    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:41:01.049252 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043437    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043440    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043443    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043447    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043450    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043463    2568 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043466    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043470    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043473    2568 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043476    2568 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043479    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043482    2568 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043485    2568 flags.go:64] FLAG: --port="10250"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043489    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043492    2568 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-03986a7e375fb731c"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043495    2568 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043498    2568 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043501    2568 flags.go:64] FLAG: --register-node="true"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043504    2568 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043508    2568 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043511    2568 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043514    2568 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043517    2568 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043521    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043524    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043527    2568 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:41:01.049896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043530    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043533    2568 flags.go:64] FLAG: --runonce="false"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043536    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043540    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043543    2568 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043546    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043549    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043552    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043555    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043562    2568 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043565    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043568    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043571    2568 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043574    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043578    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043581    2568 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043584    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043591    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043594    2568 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043598    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043603    2568 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043606    2568 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043609    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043612    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043615    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:41:01.050503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043618    2568 flags.go:64] FLAG: --v="2"
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043622    2568 flags.go:64] FLAG: --version="false"
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043627    2568 flags.go:64] FLAG: --vmodule=""
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043631    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.043634    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043800    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043805    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043809    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043813    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043816    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043819    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043822    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043825    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043828    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043830    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043833    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043840    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043842    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043845    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043849    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.051119 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043853    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043856    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043859    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043862    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043865    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043867    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043870    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043873    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043876    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043878    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043881    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043884    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043886    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043889    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043891    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043894    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043896    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043898    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043901    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.051645 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043904    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043906    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043909    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043911    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043915    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043918    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043921    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043923    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043926    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043931    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043935    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043938    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043941    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043944    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043946    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043949    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043952    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043955    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043957    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043960    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.052216 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043962    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043965    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043968    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043971    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043973    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043976    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043978    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043981    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043983    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043986    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043989    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.043991    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044001    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044004    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044007    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044009    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044012    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044015    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044018    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.052693 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044021    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044023    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044029    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044032    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044035    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044037    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044040    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044042    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044045    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044048    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044051    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044054    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044056    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044059    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044062    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044065    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044067    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044070    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044072    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044076    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.053151 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044078    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044081    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044083    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044086    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044088    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044091    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044093    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044096    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044099    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044101    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044104    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044107    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044110    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.044140    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044184    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044192    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.053634 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044195    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044198    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044200    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044203    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044206    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044208    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044211    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044214    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044216    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044219    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044221    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044224    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044226    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044229    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044232    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044234    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044236    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044239    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044241    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.054026 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044244    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044246    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044250    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044254    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044257    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044260    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044262    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044265    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044268    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044271    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044274    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044278    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044281    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044284    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044287    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044290    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044292    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044295    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044298    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.054463 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044300    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044303    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044305    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044307    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044310    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044312    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044315    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044318    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044320    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044323    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044325    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044328    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044331    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044333    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044336    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044338    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044341    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044343    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044346    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.054934 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044348    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044351    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044354    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044356    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044359    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044363    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044367    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044370    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044373    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044375    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044378    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044380    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044383    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044385    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044388    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044391    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044393    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044396    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044398    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044401    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.055374 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044404    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044406    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044409    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044411    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044414    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044417    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044420    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044422    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044425    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044428    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044430    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044433    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044435    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044437    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044440    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044443    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044445    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044450    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044454    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.055871 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044456    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044461    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044465    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044468    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044470    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044473    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044476    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.044479    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.051528    2568 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.051546    2568 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.052175    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.052182    2568 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052250    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052256    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052259    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052262    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052265    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052267    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052270    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.056341 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052273    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052276    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052279    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052282    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052285    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052288    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052290    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052293    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052296    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052298    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052301    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052304    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052308    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052311    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052314    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052316    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052319    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052322    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052325    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.056803 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052327    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052330    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052333    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052337    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052340    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052342    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052345    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052347    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052350    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052353    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052355    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052358    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052361    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052363    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052366    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052368    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052372    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052374    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052377    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.057271 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052380    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052382    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052385    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052388    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052390    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052393    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052396    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052399    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052401    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052404    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052407    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052409    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052412    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052414    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052417    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052420    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052422    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052425    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052428    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.057709 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052431    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052434    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052436    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052439    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052441    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052445    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052450    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052453    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052456    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052458    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052461    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052464    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052467    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052469    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052472    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052475    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052477    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052480    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052483    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.058174 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052488    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052491    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052495    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052498    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052501    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052504    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052507    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052510    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052513    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052516    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052518    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052521    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052525    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052528    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052530    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052533    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052535    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052538    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052540    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052543    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052545    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.058615 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052548    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052550    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.052581    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052733    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052738    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052742    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052746    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052749    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052752    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052755    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052758    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052760    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052763    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052766    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052779    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052782    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.059156 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052785    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052788    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052790    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052795    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052798    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052802    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052806    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052808    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052812    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052814    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052817    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052820    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052822    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052825    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052827    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052830    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052833    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052836    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052838    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.059543 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052840    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052843    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052845    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052848    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052851    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052853    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052856    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052859    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052861    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052864    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052866    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052869    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052871    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052874    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052876    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052879    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052884    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052886    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052889    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052891    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.060208 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052894    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052896    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052900    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052902    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052905    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052907    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052910    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052912    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052915    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052918    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052921    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052923    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052926    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052928    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052930    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052934    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052937    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052940    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052942    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.060906 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052945    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052948    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052950    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052953    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052955    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052958    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052960    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052963    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052966    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052969    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052972    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052975    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052977    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052979    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052982    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052985    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052988    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052990    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052993    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052995    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.061428 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.052998    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053001    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053003    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053006    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053008    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053011    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053014    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053016    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053020    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053023    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053025    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053028    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053030    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053033    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053036    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.053061    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:01.061920 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053095    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053099    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053102    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053104    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053107    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053111    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053114    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053117    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053119    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053122    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053125    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053127    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053130    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053133    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053136    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053139    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053142    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053144    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053147    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:01.062457 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053149    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053151    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053154    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053157    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053160    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053163    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053167    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053170    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053172    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053175    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053178    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053180    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053183    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053186    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053188    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053191    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053193    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053196    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053200    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053202    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:01.063152 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053205    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053207    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053210    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053212    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053215    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053218    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053220    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053223    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053226    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053229    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053231    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053234    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053236    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053239    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053241    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053244    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053246    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053249    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053253    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:01.063646 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053257    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053259    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053263    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053267    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053269    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053272    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053275    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053278    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053280    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053283    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053286    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053289    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053292    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053294    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053297    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053299    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053304    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053308    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053311    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053314    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:01.064207 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053316    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053320    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053322    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053325    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053327    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053330    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053332    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053335    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053349    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053353    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053357    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053360    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053363    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053366    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053368    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053371    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053374    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053376    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053379    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:01.064845 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053381    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053384    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053387    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053389    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053393    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053396    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053398    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053401    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:01.053403    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.053465    2568 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.055746    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.056707    2568 server.go:1014] "Starting client certificate rotation"
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.056793    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.056821    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.058607    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:41:01.065297 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.059811    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:41:01.070765 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.070744    2568 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:41:01.073583 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.073571    2568 log.go:25] "Validated CRI v1 image API"
Oct 30 19:41:01.074371 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.074350    2568 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:41:01.078757 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.078739    2568 fs.go:135] Filesystem UUIDs: map[1d5b3fd0-82d8-43f5-aea4-b9746774141b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 99d03b13-6faa-4eb3-aa8f-e0e517b6730b:/dev/nvme0n1p3]
Oct 30 19:41:01.078824 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.078757    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:41:01.085189 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.085071    2568 manager.go:217] Machine: {Timestamp:2026-10-30 19:41:01.083663875 +0000 UTC m=+0.338508404 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109517 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274b4129928ac65ddd82fd88fdc727 SystemUUID:ec274b41-2992-8ac6-5ddd-82fd88fdc727 BootID:64b9989a-a8cf-4083-b8ed-02a9545e5b9e Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:33:03:4a:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:33:03:4a:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:9e:04:48:95:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:41:01.085189 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.085181    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:41:01.085315 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.085259    2568 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:41:01.087918 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.087896    2568 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:41:01.088054 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.087919    2568 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-27.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:41:01.088095 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.088056    2568 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:41:01.088095 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.088065    2568 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:41:01.088150 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.088105    2568 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:41:01.088150 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.088123    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:41:01.089203 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.089194    2568 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:41:01.093277 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.093265    2568 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:41:01.093390 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.093381    2568 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:41:01.094125 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.094116    2568 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:41:01.094156 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.094128    2568 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:41:01.094156 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.094137    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:41:01.094156 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.094147    2568 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:41:01.094156 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.094156    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:41:01.094617 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.094606    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:41:01.096892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.096875    2568 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:41:01.103739 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.103719    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:41:01.104678 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.104666    2568 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:41:01.104715 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.104696    2568 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:41:01.105760 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105749    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:41:01.105806 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105766    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:41:01.105806 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105782    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:41:01.105806 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105787    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:41:01.105806 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105794    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:41:01.105806 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105800    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:41:01.105806 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105808    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:41:01.105948 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105813    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:41:01.105948 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105821    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:41:01.105948 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105826    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:41:01.105948 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105840    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:41:01.105948 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.105850    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:41:01.107226 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.107217    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:41:01.107226 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.107226    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:41:01.111077 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.111063    2568 server.go:1263] "Started kubelet"
Oct 30 19:41:01.111204 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.111147    2568 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:41:01.111436 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.111395    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:41:01.111476 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.111456    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:41:01.111954 ip-10-0-133-27 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:41:01.112896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.112831    2568 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:41:01.112896 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.112913    2568 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:41:01.118819 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.118791    2568 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:41:01.119656 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.119639    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:41:01.121493 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.121471    2568 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:41:01.121589 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.121565    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found"
Oct 30 19:41:01.122466 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122447    2568 factory.go:55] Registering systemd factory
Oct 30 19:41:01.122466 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122468    2568 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:41:01.122601 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122494    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:41:01.122601 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122495    2568 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:41:01.122601 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122522    2568 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:41:01.122727 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122671    2568 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:41:01.122727 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122681    2568 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:41:01.122727 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122706    2568 factory.go:153] Registering CRI-O factory
Oct 30 19:41:01.122727 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122716    2568 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:41:01.122858 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122756    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:41:01.122858 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122787    2568 factory.go:103] Registering Raw factory
Oct 30 19:41:01.122858 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.122799    2568 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:41:01.124489 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.124445    2568 manager.go:319] Starting recovery of all containers
Oct 30 19:41:01.137543 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.137361    2568 manager.go:324] Recovery completed
Oct 30 19:41:01.141532 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.141520    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.143505 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.143480    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.143568 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.143553    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.143600 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.143573    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.144177 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.144162    2568 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:41:01.144177 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.144174    2568 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:41:01.144281 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.144194    2568 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:41:01.144281 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.144209    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.145195 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.145179    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.145241 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.145208    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.145241 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.145218    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.146441 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.146428    2568 policy_none.go:50] "Start"
Oct 30 19:41:01.146520 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.146447    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:41:01.146520 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.146460    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:41:01.148628 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.148616    2568 policy_none.go:44] "Start"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.185560    2568 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189179    2568 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.189209    2568 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189218    2568 server.go:87] "Starting device plugin registration server"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189484    2568 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189496    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189580    2568 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189667    2568 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.189674    2568 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.190438    2568 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.190488    2568 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-27.us-west-1.compute.internal\" not found"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.192009    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.193166    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.193184    2568 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.193202    2568 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:41:01.200235 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.193232    2568 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:41:01.291190 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.291101    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.292066 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.292047    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.292167 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.292079    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.292167 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.292094    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.292167 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.292123    2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.293282 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.293263    2568 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"]
Oct 30 19:41:01.293369 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.293318    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.294039 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.294024    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.294121 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.294054    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.294121 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.294068    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.296231 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296216    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.296365 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296351    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.296404 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296380    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.296867 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296851    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.296867 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296862    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.297006 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296883    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.297006 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296886    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.297006 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296892    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.297006 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.296898    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.298926 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.298911    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.298983 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.298944    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.299593 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.299579    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.299640 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.299604    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.299640 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.299614    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.312673 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.312628    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-27.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:41:01.312764 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.312719    2568 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-27.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:41:01.313464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.312622    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-27.us-west-1.compute.internal.18e36607fc858b60  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-27.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-27.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:01.111004 +0000 UTC m=+0.365848530,LastTimestamp:2026-10-30 19:41:01.111004 +0000 UTC m=+0.365848530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-27.us-west-1.compute.internal,}"
Oct 30 19:41:01.319132 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.319108    2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.323918 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.323901    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d870f2d14f468b90253c9dd98ce2b16-config\") pod \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"5d870f2d14f468b90253c9dd98ce2b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.324004 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.323925    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.324004 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.323944    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.325714 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.325699    2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.345361 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.345338    2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:41:01.345456 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.345387    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b68v2"
Oct 30 19:41:01.351647 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.351628    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b68v2"
Oct 30 19:41:01.424146 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.424119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d870f2d14f468b90253c9dd98ce2b16-config\") pod \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"5d870f2d14f468b90253c9dd98ce2b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.424146 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.424148    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.424312 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.424164    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.424312 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.424203    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.424312 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.424207    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d870f2d14f468b90253c9dd98ce2b16-config\") pod \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"5d870f2d14f468b90253c9dd98ce2b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.424312 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.424279    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.507079 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.507048    2568 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:01.556322 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.556217    2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:01.604829 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.604799    2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:01.604829 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.604822    2568 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-27.us-west-1.compute.internal\" not found" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.606026 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.606005    2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:01.622840 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.622823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.631939 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.631926    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.656912 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:01.656891    2568 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-27.us-west-1.compute.internal\" not found" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.805112 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.805081    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:01.806211 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.806195    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:01.806298 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.806224    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:01.806298 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.806238    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:01.806298 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.806271    2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.863963 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.863891    2568 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.921938 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.921908    2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:01.994384 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.994357    2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:01.995068 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:01.995055    2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:02.056251 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.056225    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:41:02.056949 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.056347    2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:41:02.056949 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.056386    2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:41:02.056949 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.056387    2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:41:02.056949 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.056445    2568 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a8a25ac9f0dfc4e67bf96bf3051bc6f6-5b74f0dd39b75e91.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.133.27:60198->18.227.17.225:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:02.056949 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.056424    2568 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a8a25ac9f0dfc4e67bf96bf3051bc6f6-5b74f0dd39b75e91.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-27.us-west-1.compute.internal.18e36607fe75d850\": read tcp 10.0.133.27:60198->18.227.17.225:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-27.us-west-1.compute.internal.18e36607fe75d850  default   4689 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-27.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-27.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-27.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:01 +0000 UTC,LastTimestamp:2026-10-30 19:41:01.292064111 +0000 UTC m=+0.546908642,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-27.us-west-1.compute.internal,}"
Oct 30 19:41:02.094320 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.094298    2568 apiserver.go:52] "Watching apiserver"
Oct 30 19:41:02.120567 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.120501    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:41:02.357857 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.357813    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:36:02 +0000 UTC" deadline="2028-04-10 01:46:01.399451684 +0000 UTC"
Oct 30 19:41:02.357857 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.357849    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12654h4m59.041607854s"
Oct 30 19:41:02.397503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.397434    2568 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:41:02.397743 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.397726    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-cn92p","kube-system/konnectivity-agent-tznqp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7","openshift-network-diagnostics/network-check-target-pt4mt","kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-mrdzz","openshift-multus/multus-additional-cni-plugins-lngvb","openshift-network-operator/iptables-alerter-d88b4","openshift-multus/network-metrics-daemon-tfswf","openshift-ovn-kubernetes/ovnkube-node-4qcq7","openshift-image-registry/node-ca-t7wjp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"]
Oct 30 19:41:02.400290 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.400274    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.402545 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.402518    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.404700 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.404684    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.406803 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.406788    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:02.406893 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.406846    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:02.409086 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.409068    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.411353 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.411331    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.413597 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.413581    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.417193 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.417177    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:02.417280 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.417251    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:02.419695 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.419662    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.422007 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.421987    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.423601 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.423586    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429659    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-node-log\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429762    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429811    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysconfig\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429875    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-sys\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429914    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429959    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.429995    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-kubernetes\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430023    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49dc5027-2bf7-4365-a107-6dc6bf106382-serviceca\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430054    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-script-lib\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430120    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-kubelet\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430149    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-systemd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430203    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d555bfc-7445-4727-a110-ce66dae3ee49-ovn-node-metrics-cert\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-registration-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.431630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430271    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-tuned\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430300    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/ae767808-d262-4e49-8754-7149f5ec3fd1-kube-api-access-hrrlg\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430325    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-host-slash\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-kubelet\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430384    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-log-socket\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkc2\" (UniqueName: \"kubernetes.io/projected/4d555bfc-7445-4727-a110-ce66dae3ee49-kube-api-access-grkc2\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-host\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngw5v\" (UniqueName: \"kubernetes.io/projected/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-kube-api-access-ngw5v\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430494    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430524    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v468k\" (UniqueName: \"kubernetes.io/projected/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-kube-api-access-v468k\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430555    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9ct\" (UniqueName: \"kubernetes.io/projected/eb6f0ca7-6e21-4447-a036-2c91f24e885f-kube-api-access-9m9ct\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-var-lib-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430608    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-tmp\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430638    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cnibin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-device-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430701    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4mt\" (UniqueName: \"kubernetes.io/projected/49dc5027-2bf7-4365-a107-6dc6bf106382-kube-api-access-rc4mt\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.432435 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-daemon-config\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430760    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-systemd-units\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430795    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-sys-fs\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.430834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89kj\" (UniqueName: \"kubernetes.io/projected/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kube-api-access-x89kj\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxjr\" (UniqueName: \"kubernetes.io/projected/0f76c795-db38-4b17-b2d5-c3c3d9905edc-kube-api-access-fwxjr\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431233    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-hostroot\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431264    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f1128e1-21b4-437a-a7f5-e94567bec795-konnectivity-ca\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431295    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-etc-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431322    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-multus-certs\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-run\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431386    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-system-cni-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431416    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-lib-modules\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cnibin\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431510    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431542    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.433031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431573    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cni-binary-copy\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.431653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-multus\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-slash\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432527    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-env-overrides\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432565    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49dc5027-2bf7-4365-a107-6dc6bf106382-host\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-k8s-cni-cncf-io\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432697    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-conf-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432743    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f1128e1-21b4-437a-a7f5-e94567bec795-agent-certs\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432804    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-socket-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432843    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-modprobe-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-conf\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432912    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432952    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-socket-dir-parent\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.432984    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-netns\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433029    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.433613 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433059    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-systemd\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433087    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-var-lib-kubelet\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-iptables-alerter-script\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-bin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433185    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-etc-kubernetes\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-ovn\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433252    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-system-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433303    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-netns\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433340    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-bin\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-netd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433426    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-config\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.434111 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.433469    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.471999 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.471968    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:41:02.471999 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.471974    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:41:02.472134 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.472014    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6h9\""
Oct 30 19:41:02.472134 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.472022    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.472134 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.472050    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pcf9z\""
Oct 30 19:41:02.472436 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.472423    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:41:02.473649 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.473636    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.476614 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.476594    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:41:02.476614 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.476608    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.478798 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.478775    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.478897 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.478835    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:41:02.478897 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.478876    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bk6w\""
Oct 30 19:41:02.526917 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.526896    2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:41:02.527007 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.526960    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gqj5b"
Oct 30 19:41:02.534178 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-multus-certs\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.534243 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534183    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-run\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.534243 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534212    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-system-cni-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534243 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534231    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.534341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534265    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-multus-certs\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.534341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534274    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-run\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.534341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534287    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-system-cni-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534432 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534336    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-lib-modules\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.534432 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534357    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.534432 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534366    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cnibin\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534432 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534394    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534432 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534396    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-lib-modules\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.534432 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534420    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cnibin\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534420    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534472    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cni-binary-copy\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534491    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-multus\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-slash\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534528    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534538    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-env-overrides\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534539    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-multus\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534564    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-slash\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49dc5027-2bf7-4365-a107-6dc6bf106382-host\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534591    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49dc5027-2bf7-4365-a107-6dc6bf106382-host\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.534620 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534596    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-k8s-cni-cncf-io\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534627    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-conf-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534650    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-conf-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f1128e1-21b4-437a-a7f5-e94567bec795-agent-certs\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534627    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-k8s-cni-cncf-io\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534681    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-socket-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534698    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-modprobe-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534713    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-conf\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534729    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-socket-dir-parent\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534830    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-netns\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534840    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-modprobe-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534860    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534886    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534889    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-conf\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534907    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-socket-dir-parent\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534912    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-systemd\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534941    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534952    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-var-lib-kubelet\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534965    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-systemd\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534981    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-iptables-alerter-script\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.534988    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-netns\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535005    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-bin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535027    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-var-lib-kubelet\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-etc-kubernetes\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535055    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535061    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-ovn\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535087    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-system-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535089    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-etc-kubernetes\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535088    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-bin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535129    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-ovn\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535107    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-socket-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535147    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-system-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535159    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-netns\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535164    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cni-binary-copy\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.535892 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-bin\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535161    2568 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535208    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-netns\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535223    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-netd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535240    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-bin\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535249    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-config\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535269    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-netd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535278    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-node-log\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535343    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-node-log\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysconfig\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535391    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535399    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535405    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-sys\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535421    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysconfig\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535433    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535455    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:02.536863 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535459    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-sys\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535474    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-kubernetes\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535490    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535505    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49dc5027-2bf7-4365-a107-6dc6bf106382-serviceca\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535531    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-kubernetes\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535529    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535557    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-script-lib\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535567    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535582    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-kubelet\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535607    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535624    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-systemd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535632    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-kubelet\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535641    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d555bfc-7445-4727-a110-ce66dae3ee49-ovn-node-metrics-cert\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-registration-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-tuned\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535678    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-systemd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535695    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/ae767808-d262-4e49-8754-7149f5ec3fd1-kube-api-access-hrrlg\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.535723    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:02.537654 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535733    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-registration-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535763    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-host-slash\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.535829    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-host-slash\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.536312    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:03.035776061 +0000 UTC m=+2.290620588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-kubelet\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536379    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-log-socket\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536360    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-kubelet\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536408    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkc2\" (UniqueName: \"kubernetes.io/projected/4d555bfc-7445-4727-a110-ce66dae3ee49-kube-api-access-grkc2\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536435    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-host\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536442    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-log-socket\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw5v\" (UniqueName: \"kubernetes.io/projected/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-kube-api-access-ngw5v\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536497    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536510    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-host\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536603    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v468k\" (UniqueName: \"kubernetes.io/projected/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-kube-api-access-v468k\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536633    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9ct\" (UniqueName: \"kubernetes.io/projected/eb6f0ca7-6e21-4447-a036-2c91f24e885f-kube-api-access-9m9ct\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536645    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-var-lib-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538257 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536684    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-tmp\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536707    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-var-lib-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536710    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cnibin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536755    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-device-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536796    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4mt\" (UniqueName: \"kubernetes.io/projected/49dc5027-2bf7-4365-a107-6dc6bf106382-kube-api-access-rc4mt\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536796    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cnibin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536840    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-device-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536910    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-daemon-config\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536940    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-systemd-units\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.536988    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-sys-fs\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537016    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89kj\" (UniqueName: \"kubernetes.io/projected/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kube-api-access-x89kj\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537050    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-systemd-units\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537057    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxjr\" (UniqueName: \"kubernetes.io/projected/0f76c795-db38-4b17-b2d5-c3c3d9905edc-kube-api-access-fwxjr\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537057    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-sys-fs\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537096    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-hostroot\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537126    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f1128e1-21b4-437a-a7f5-e94567bec795-konnectivity-ca\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-etc-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.538805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537224    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-etc-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.539351 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537244    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-hostroot\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.539351 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537408    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-daemon-config\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.539351 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.537810    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f1128e1-21b4-437a-a7f5-e94567bec795-konnectivity-ca\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.539351 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.538526    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-tuned\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.539351 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.538801    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f1128e1-21b4-437a-a7f5-e94567bec795-agent-certs\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.539351 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.539042    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-tmp\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.558508 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.558485    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-57p4t"]
Oct 30 19:41:02.561411 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.561395    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x6kbk"]
Oct 30 19:41:02.561538 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.561521    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.564260 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.564244    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.564321 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.564299    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:02.574211 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.574185    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:41:02.574456 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.574441    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:41:02.575037 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575003    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:41:02.575037 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575010    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsfbm\""
Oct 30 19:41:02.575187 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575095    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.575187 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575178    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-env-overrides\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.575426 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575360    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:41:02.575426 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575403    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47f99\""
Oct 30 19:41:02.575426 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575406    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:41:02.575588 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575431    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.575588 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575447    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.575747 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575723    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-config\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.575870 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575851    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhxv\""
Oct 30 19:41:02.575930 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575905    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.575992 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.575978    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.576067 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.576052    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49dc5027-2bf7-4365-a107-6dc6bf106382-serviceca\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.581696 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.581680    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:41:02.586330 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.586310    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.586411 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.586357    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-iptables-alerter-script\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.588156 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.588136    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d555bfc-7445-4727-a110-ce66dae3ee49-ovn-node-metrics-cert\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.600300 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.600273    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:41:02.600409 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.600327    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.600409 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.600384    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.602042 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.601990    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.602154 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.602043    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shc6v\""
Oct 30 19:41:02.605045 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.605023    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjk7j\""
Oct 30 19:41:02.605045 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.605037    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:41:02.606646 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.606627    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-script-lib\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.612071 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.612053    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:02.612071 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.612072    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:02.612184 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.612082    2568 projected.go:196] Error preparing data for projected volume kube-api-access-7695w for pod openshift-network-diagnostics/network-check-target-pt4mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:02.612184 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.612168    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w podName:d94a8213-30da-4eef-9967-8afc5725b7db nodeName:}" failed. No retries permitted until 2026-10-30 19:41:03.112150232 +0000 UTC m=+2.366994768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7695w" (UniqueName: "kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w") pod "network-check-target-pt4mt" (UID: "d94a8213-30da-4eef-9967-8afc5725b7db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:02.615046 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.615020    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.616986 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.616961    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/ae767808-d262-4e49-8754-7149f5ec3fd1-kube-api-access-hrrlg\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.617206 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.617188    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkc2\" (UniqueName: \"kubernetes.io/projected/4d555bfc-7445-4727-a110-ce66dae3ee49-kube-api-access-grkc2\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.617995 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.617975    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw5v\" (UniqueName: \"kubernetes.io/projected/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-kube-api-access-ngw5v\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.618946 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.618927    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4mt\" (UniqueName: \"kubernetes.io/projected/49dc5027-2bf7-4365-a107-6dc6bf106382-kube-api-access-rc4mt\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.632245 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.632224    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89kj\" (UniqueName: \"kubernetes.io/projected/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kube-api-access-x89kj\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.632332 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.632289    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9ct\" (UniqueName: \"kubernetes.io/projected/eb6f0ca7-6e21-4447-a036-2c91f24e885f-kube-api-access-9m9ct\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:02.633326 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.633304    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v468k\" (UniqueName: \"kubernetes.io/projected/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-kube-api-access-v468k\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.633403 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.633330    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxjr\" (UniqueName: \"kubernetes.io/projected/0f76c795-db38-4b17-b2d5-c3c3d9905edc-kube-api-access-fwxjr\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.637813 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.637793    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69l5\" (UniqueName: \"kubernetes.io/projected/56510de8-5bb1-45b5-a057-f83143d0d836-kube-api-access-b69l5\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.637897 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.637837    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-dbus\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.637897 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.637863    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56510de8-5bb1-45b5-a057-f83143d0d836-tmp-dir\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.638084 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.638066    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56510de8-5bb1-45b5-a057-f83143d0d836-hosts-file\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.638134 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.638095    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-kubelet-config\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.638317 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.638287    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.683118 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.681641    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:41:02.683118 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.681688    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqb6z\""
Oct 30 19:41:02.683118 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.681723    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:41:02.723918 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.723891    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cn92p"
Oct 30 19:41:02.738953 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.738925    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69l5\" (UniqueName: \"kubernetes.io/projected/56510de8-5bb1-45b5-a057-f83143d0d836-kube-api-access-b69l5\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.739088 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.738963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-dbus\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.739088 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.738995    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56510de8-5bb1-45b5-a057-f83143d0d836-tmp-dir\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.739088 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739029    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56510de8-5bb1-45b5-a057-f83143d0d836-hosts-file\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.739088 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-kubelet-config\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.739327 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739149    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.739327 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739156    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-dbus\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.739327 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739161    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56510de8-5bb1-45b5-a057-f83143d0d836-hosts-file\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.739327 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739188    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-kubelet-config\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:02.739327 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.739264    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:02.739495 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:02.739333    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret podName:710b36be-68a9-4ed5-84c6-30d5e3dc6801 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:03.239313326 +0000 UTC m=+2.494157860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret") pod "global-pull-secret-syncer-x6kbk" (UID: "710b36be-68a9-4ed5-84c6-30d5e3dc6801") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:02.739495 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.739343    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56510de8-5bb1-45b5-a057-f83143d0d836-tmp-dir\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.752017 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.751992    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:02.769045 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.769022    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:02.773520 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.773503    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:02.784995 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.784975    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:02.788147 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.788132    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:02.806709 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.806690    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69l5\" (UniqueName: \"kubernetes.io/projected/56510de8-5bb1-45b5-a057-f83143d0d836-kube-api-access-b69l5\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.809112 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.809077    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae767808_d262_4e49_8754_7149f5ec3fd1.slice/crio-685c0f46dc78a9d1ca7fd7ff1d995d62da807d58884cfb7cd1661cc263a92995 WatchSource:0}: Error finding container 685c0f46dc78a9d1ca7fd7ff1d995d62da807d58884cfb7cd1661cc263a92995: Status 404 returned error can't find the container with id 685c0f46dc78a9d1ca7fd7ff1d995d62da807d58884cfb7cd1661cc263a92995
Oct 30 19:41:02.809581 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.809561    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1128e1_21b4_437a_a7f5_e94567bec795.slice/crio-1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06 WatchSource:0}: Error finding container 1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06: Status 404 returned error can't find the container with id 1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06
Oct 30 19:41:02.811226 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.811202    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8723c555bc23b9ece9590b4168888e4f.slice/crio-3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b WatchSource:0}: Error finding container 3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b: Status 404 returned error can't find the container with id 3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b
Oct 30 19:41:02.811938 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.811917    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a32270_0ad2_4fb4_b15f_7dfcfb915a3d.slice/crio-f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4 WatchSource:0}: Error finding container f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4: Status 404 returned error can't find the container with id f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4
Oct 30 19:41:02.813220 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.813203    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f76c795_db38_4b17_b2d5_c3c3d9905edc.slice/crio-6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42 WatchSource:0}: Error finding container 6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42: Status 404 returned error can't find the container with id 6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42
Oct 30 19:41:02.815398 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.815350    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:41:02.816500 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.816476    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4b078f_2e4a_4c6c_a352_16e2a2912bcb.slice/crio-9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432 WatchSource:0}: Error finding container 9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432: Status 404 returned error can't find the container with id 9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432
Oct 30 19:41:02.817075 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.817054    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d870f2d14f468b90253c9dd98ce2b16.slice/crio-2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913 WatchSource:0}: Error finding container 2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913: Status 404 returned error can't find the container with id 2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913
Oct 30 19:41:02.831379 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.831360    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:02.838996 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.838976    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:02.840858 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.840824    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49dc5027_2bf7_4365_a107_6dc6bf106382.slice/crio-7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb WatchSource:0}: Error finding container 7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb: Status 404 returned error can't find the container with id 7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb
Oct 30 19:41:02.847108 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.847087    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d555bfc_7445_4727_a110_ce66dae3ee49.slice/crio-807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad WatchSource:0}: Error finding container 807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad: Status 404 returned error can't find the container with id 807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad
Oct 30 19:41:02.882508 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:02.882488    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:02.891682 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:02.891660    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56510de8_5bb1_45b5_a057_f83143d0d836.slice/crio-beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d WatchSource:0}: Error finding container beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d: Status 404 returned error can't find the container with id beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d
Oct 30 19:41:03.041400 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.041315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:03.041541 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.041444    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:03.041541 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.041508    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:04.041491363 +0000 UTC m=+3.296335883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:03.141677 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.141639    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:03.142333 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.141817    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:03.142333 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.141838    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:03.142333 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.141851    2568 projected.go:196] Error preparing data for projected volume kube-api-access-7695w for pod openshift-network-diagnostics/network-check-target-pt4mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:03.142333 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.141906    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w podName:d94a8213-30da-4eef-9967-8afc5725b7db nodeName:}" failed. No retries permitted until 2026-10-30 19:41:04.141889733 +0000 UTC m=+3.396734252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7695w" (UniqueName: "kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w") pod "network-check-target-pt4mt" (UID: "d94a8213-30da-4eef-9967-8afc5725b7db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:03.197768 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.197717    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"5d870f2d14f468b90253c9dd98ce2b16","Type":"ContainerStarted","Data":"2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913"}
Oct 30 19:41:03.198852 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.198828    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerStarted","Data":"6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42"}
Oct 30 19:41:03.199933 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.199910    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrdzz" event={"ID":"ae767808-d262-4e49-8754-7149f5ec3fd1","Type":"ContainerStarted","Data":"685c0f46dc78a9d1ca7fd7ff1d995d62da807d58884cfb7cd1661cc263a92995"}
Oct 30 19:41:03.200885 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.200865    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7" event={"ID":"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d","Type":"ContainerStarted","Data":"f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4"}
Oct 30 19:41:03.201924 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.201902    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tznqp" event={"ID":"4f1128e1-21b4-437a-a7f5-e94567bec795","Type":"ContainerStarted","Data":"1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06"}
Oct 30 19:41:03.203003 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.202983    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57p4t" event={"ID":"56510de8-5bb1-45b5-a057-f83143d0d836","Type":"ContainerStarted","Data":"beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d"}
Oct 30 19:41:03.204092 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.204070    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad"}
Oct 30 19:41:03.205145 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.205125    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t7wjp" event={"ID":"49dc5027-2bf7-4365-a107-6dc6bf106382","Type":"ContainerStarted","Data":"7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb"}
Oct 30 19:41:03.206248 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.206226    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d88b4" event={"ID":"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb","Type":"ContainerStarted","Data":"9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432"}
Oct 30 19:41:03.207199 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.207179    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"8723c555bc23b9ece9590b4168888e4f","Type":"ContainerStarted","Data":"3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b"}
Oct 30 19:41:03.242433 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.242405    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:03.242571 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.242554    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:03.242626 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:03.242615    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret podName:710b36be-68a9-4ed5-84c6-30d5e3dc6801 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:04.242598505 +0000 UTC m=+3.497443024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret") pod "global-pull-secret-syncer-x6kbk" (UID: "710b36be-68a9-4ed5-84c6-30d5e3dc6801") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:03.282152 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.282128    2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:03.529971 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.529907    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:03 +0000 UTC" deadline="2028-05-29 13:42:35.853915282 +0000 UTC"
Oct 30 19:41:03.529971 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.529961    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13842h1m32.323956426s"
Oct 30 19:41:03.533625 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.533604    2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:03.608960 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:03.608923    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae9ccea_c7c3_4421_aad0_f36dd1cbc8e3.slice/crio-0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92 WatchSource:0}: Error finding container 0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92: Status 404 returned error can't find the container with id 0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92
Oct 30 19:41:03.657397 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:03.657368    2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:04.051160 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.051120    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:04.051351 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.051288    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:04.051425 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.051355    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:06.051336549 +0000 UTC m=+5.306181072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:04.152471 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.152427    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:04.152988 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.152616    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:04.152988 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.152640    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:04.152988 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.152653    2568 projected.go:196] Error preparing data for projected volume kube-api-access-7695w for pod openshift-network-diagnostics/network-check-target-pt4mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:04.152988 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.152710    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w podName:d94a8213-30da-4eef-9967-8afc5725b7db nodeName:}" failed. No retries permitted until 2026-10-30 19:41:06.152692259 +0000 UTC m=+5.407536792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7695w" (UniqueName: "kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w") pod "network-check-target-pt4mt" (UID: "d94a8213-30da-4eef-9967-8afc5725b7db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:04.195705 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.195215    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:04.195705 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.195353    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:04.195705 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.195439    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:04.195705 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.195511    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:04.195705 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.195575    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:04.195705 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.195661    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:04.233175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.233016    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cn92p" event={"ID":"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3","Type":"ContainerStarted","Data":"0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92"}
Oct 30 19:41:04.253529 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.253362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:04.253529 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.253492    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:04.253752 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:04.253550    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret podName:710b36be-68a9-4ed5-84c6-30d5e3dc6801 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:06.253530677 +0000 UTC m=+5.508375196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret") pod "global-pull-secret-syncer-x6kbk" (UID: "710b36be-68a9-4ed5-84c6-30d5e3dc6801") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:04.530797 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.530710    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:03 +0000 UTC" deadline="2028-06-25 06:17:39.896477908 +0000 UTC"
Oct 30 19:41:04.530797 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:04.530751    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14482h36m35.365731198s"
Oct 30 19:41:06.073682 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:06.073408    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:06.074218 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.073728    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:06.074218 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.073886    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:10.073861407 +0000 UTC m=+9.328705930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:06.175939 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:06.175090    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:06.175939 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.175267    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:06.175939 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.175286    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:06.175939 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.175298    2568 projected.go:196] Error preparing data for projected volume kube-api-access-7695w for pod openshift-network-diagnostics/network-check-target-pt4mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:06.175939 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.175369    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w podName:d94a8213-30da-4eef-9967-8afc5725b7db nodeName:}" failed. No retries permitted until 2026-10-30 19:41:10.17534962 +0000 UTC m=+9.430194145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7695w" (UniqueName: "kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w") pod "network-check-target-pt4mt" (UID: "d94a8213-30da-4eef-9967-8afc5725b7db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:06.194407 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:06.194340    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:06.194582 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.194479    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:06.194582 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:06.194550    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:06.194725 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.194617    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:06.194725 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:06.194666    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:06.194835 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.194747    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:06.276313 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:06.276237    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:06.276611 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.276409    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:06.276611 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:06.276476    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret podName:710b36be-68a9-4ed5-84c6-30d5e3dc6801 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:10.276457446 +0000 UTC m=+9.531301970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret") pod "global-pull-secret-syncer-x6kbk" (UID: "710b36be-68a9-4ed5-84c6-30d5e3dc6801") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:08.193431 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:08.193400    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:08.193785 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:08.193556    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:08.193785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:08.193633    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:08.193785 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:08.193708    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:08.193785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:08.193759    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:08.193948 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:08.193849    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:08.281741 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:08.281683    2568 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="252153c5cc5df9f762f900957795cfd6da543107574021f355bdfe53eb459bdd" exitCode=0
Oct 30 19:41:08.281939 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:08.281746    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerDied","Data":"252153c5cc5df9f762f900957795cfd6da543107574021f355bdfe53eb459bdd"}
Oct 30 19:41:09.296280 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:09.296239    2568 generic.go:356] "Generic (PLEG): container finished" podID="8723c555bc23b9ece9590b4168888e4f" containerID="f3d4ad7b52e3754b68d512dc0d15bf95f5756a1dd350a86d0b7fc23d7fbfa194" exitCode=0
Oct 30 19:41:09.296772 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:09.296548    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"8723c555bc23b9ece9590b4168888e4f","Type":"ContainerDied","Data":"f3d4ad7b52e3754b68d512dc0d15bf95f5756a1dd350a86d0b7fc23d7fbfa194"}
Oct 30 19:41:09.305652 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:09.305545    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"5d870f2d14f468b90253c9dd98ce2b16","Type":"ContainerStarted","Data":"670d1fafc73f14bbf092942c2b7cdceabd60d4a86988df8b9f6c5cdfe44d20ef"}
Oct 30 19:41:09.311859 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:09.311380    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tznqp" event={"ID":"4f1128e1-21b4-437a-a7f5-e94567bec795","Type":"ContainerStarted","Data":"e473f651ae99fbfa31ca98f0c9eb54c94f185390defe31ed7c3aea5e3b93c39e"}
Oct 30 19:41:09.688513 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:09.688406    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tznqp" podStartSLOduration=2.343386009 podStartE2EDuration="7.688388406s" totalImagesPullingTime="5.345002397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:09.561970679 +0000 UTC m=+8.816815267" watchObservedRunningTime="2026-10-30 19:41:09.688388406 +0000 UTC m=+8.943232949"
Oct 30 19:41:10.113359 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.113321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:10.113541 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.113487    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:10.113603 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.113549    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:18.113530285 +0000 UTC m=+17.368374820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:10.193552 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.193514    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:10.193552 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.193554    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:10.193776 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.193514    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:10.193776 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.193688    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:10.193776 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.193741    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:10.193975 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.193830    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:10.214107 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.214078    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:10.214319 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.214300    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:10.214368 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.214328    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:10.214368 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.214340    2568 projected.go:196] Error preparing data for projected volume kube-api-access-7695w for pod openshift-network-diagnostics/network-check-target-pt4mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:10.214438 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.214394    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w podName:d94a8213-30da-4eef-9967-8afc5725b7db nodeName:}" failed. No retries permitted until 2026-10-30 19:41:18.214376861 +0000 UTC m=+17.469221393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7695w" (UniqueName: "kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w") pod "network-check-target-pt4mt" (UID: "d94a8213-30da-4eef-9967-8afc5725b7db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:10.314871 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.314741    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:10.316107 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.315804    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:10.316107 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:10.315879    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret podName:710b36be-68a9-4ed5-84c6-30d5e3dc6801 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:18.315859114 +0000 UTC m=+17.570703639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret") pod "global-pull-secret-syncer-x6kbk" (UID: "710b36be-68a9-4ed5-84c6-30d5e3dc6801") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:10.317342 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.317225    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7" event={"ID":"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d","Type":"ContainerStarted","Data":"c49d9eccc7a0cebe2d9edc4c10c0d435dd19098f849376851202612ddd8eb6d6"}
Oct 30 19:41:10.321474 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.320924    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t7wjp" event={"ID":"49dc5027-2bf7-4365-a107-6dc6bf106382","Type":"ContainerStarted","Data":"91935fde36ccdcbff706b94550297e4f3e87d2d29bb06b8d4bc4fe46d84e2b28"}
Oct 30 19:41:10.327644 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.327240    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"8723c555bc23b9ece9590b4168888e4f","Type":"ContainerStarted","Data":"054e4178b8239862d642ab6beabb92993eafb365c7d825d2b985a10666124534"}
Oct 30 19:41:10.447978 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.447487    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal" podStartSLOduration=8.44746893 podStartE2EDuration="8.44746893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:09.687946574 +0000 UTC m=+8.942791108" watchObservedRunningTime="2026-10-30 19:41:10.44746893 +0000 UTC m=+9.702313472"
Oct 30 19:41:10.574808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:10.574689    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t7wjp" podStartSLOduration=2.246070539 podStartE2EDuration="8.574667141s" totalImagesPullingTime="6.328596602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:10.447800257 +0000 UTC m=+9.702644801" watchObservedRunningTime="2026-10-30 19:41:10.574667141 +0000 UTC m=+9.829511679"
Oct 30 19:41:12.193915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.193685    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:12.194242 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.193941    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:12.194242 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:12.194060    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:12.194242 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:12.194114    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:12.194242 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.194181    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:12.194377 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:12.194262    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:12.479210 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.479153    2568 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:41:12.878755 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.878713    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:12.879341 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.879317    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:12.997759 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:12.997704    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" podStartSLOduration=10.99768947 podStartE2EDuration="10.99768947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:10.576013947 +0000 UTC m=+9.830858496" watchObservedRunningTime="2026-10-30 19:41:12.99768947 +0000 UTC m=+12.252534009"
Oct 30 19:41:13.198109 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.198008    2568 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:41:12.479175437Z","UUID":"256aa15d-92c5-4471-a55c-361f69c5b1a3","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:41:13.199746 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.199631    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:41:13.199746 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.199664    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:41:13.336719 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.336619    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57p4t" event={"ID":"56510de8-5bb1-45b5-a057-f83143d0d836","Type":"ContainerStarted","Data":"97e44fe37383bb577353aad5b95cbf42a68e9fa5281063acdcae5ce871c42557"}
Oct 30 19:41:13.338952 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.338913    2568 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="2bc8196097c56876de66564670b8a8b16591c8062fa29617831bf4cea36d6165" exitCode=0
Oct 30 19:41:13.339074 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.339044    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerDied","Data":"2bc8196097c56876de66564670b8a8b16591c8062fa29617831bf4cea36d6165"}
Oct 30 19:41:13.341044 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.340940    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7" event={"ID":"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d","Type":"ContainerStarted","Data":"c5815294cb286ae246ee1bdcb53656ec75bf6668b8f308741fa81a6dc51f031f"}
Oct 30 19:41:13.344454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.344433    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cn92p" event={"ID":"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3","Type":"ContainerStarted","Data":"8e1b3fe9eef2257321e45d59203d8e7ffc59d1d304173a93ad1c87cbc15856e1"}
Oct 30 19:41:13.344531 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.344475    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:13.345566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.345549    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:13.580861 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.580809    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-57p4t" podStartSLOduration=1.2189344069999999 podStartE2EDuration="10.580789612s" totalImagesPullingTime="9.361855205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:13.462184101 +0000 UTC m=+12.717028647" watchObservedRunningTime="2026-10-30 19:41:13.580789612 +0000 UTC m=+12.835634148"
Oct 30 19:41:13.845540 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:13.845443    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cn92p" podStartSLOduration=2.824384834 podStartE2EDuration="11.845423044s" totalImagesPullingTime="9.02103821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:13.707603387 +0000 UTC m=+12.962447930" watchObservedRunningTime="2026-10-30 19:41:13.845423044 +0000 UTC m=+13.100267583"
Oct 30 19:41:14.194332 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:14.194295    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:14.194502 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:14.194310    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:14.194502 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:14.194445    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:14.194616 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:14.194501    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:14.194616 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:14.194556    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:14.194693 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:14.194639    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:14.347500 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:14.347463    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d88b4" event={"ID":"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb","Type":"ContainerStarted","Data":"ac5261dee453b3f169a7f29f956890fb256b11afdbfaeaa1dda3d9fd375a52ae"}
Oct 30 19:41:14.465736 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:14.465647    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d88b4" podStartSLOduration=3.129287374 podStartE2EDuration="12.465629471s" totalImagesPullingTime="9.336342097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:14.465500579 +0000 UTC m=+13.720345121" watchObservedRunningTime="2026-10-30 19:41:14.465629471 +0000 UTC m=+13.720474011"
Oct 30 19:41:15.353070 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:15.352852    2568 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="529e5c4b44b00fb2cce070e8b0472242bd4b3765fdb307f3b778cc53ab9883d3" exitCode=0
Oct 30 19:41:15.353576 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:15.352940    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerDied","Data":"529e5c4b44b00fb2cce070e8b0472242bd4b3765fdb307f3b778cc53ab9883d3"}
Oct 30 19:41:15.355300 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:15.355275    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrdzz" event={"ID":"ae767808-d262-4e49-8754-7149f5ec3fd1","Type":"ContainerStarted","Data":"b596e59538fe18189ec80910461cdb7f1f6e5fe6608156cb895b7f339392eca5"}
Oct 30 19:41:15.357645 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:15.357619    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7" event={"ID":"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d","Type":"ContainerStarted","Data":"e7172494e1de0f547161e37168b3f495816151b1faccc89352249b6808c05441"}
Oct 30 19:41:15.614052 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:15.613958    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7" podStartSLOduration=2.22317848 podStartE2EDuration="13.613941262s" totalImagesPullingTime="11.390762782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:15.613227378 +0000 UTC m=+14.868071918" watchObservedRunningTime="2026-10-30 19:41:15.613941262 +0000 UTC m=+14.868785803"
Oct 30 19:41:16.193868 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:16.193830    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:16.194067 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:16.193887    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:16.194067 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:16.194008    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:16.194067 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:16.194013    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:16.194225 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:16.194100    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:16.194225 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:16.194180    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:17.364964 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:17.364928    2568 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="b6b472d6dafa5c28d91e4772afcfeec7b1dc5559b17998a82b3145f9b5ce9eed" exitCode=0
Oct 30 19:41:17.365473 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:17.364983    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerDied","Data":"b6b472d6dafa5c28d91e4772afcfeec7b1dc5559b17998a82b3145f9b5ce9eed"}
Oct 30 19:41:17.543697 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:17.543642    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mrdzz" podStartSLOduration=3.523626558 podStartE2EDuration="15.543623462s" totalImagesPullingTime="12.019996904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:15.734348372 +0000 UTC m=+14.989192912" watchObservedRunningTime="2026-10-30 19:41:17.543623462 +0000 UTC m=+16.798468003"
Oct 30 19:41:18.187025 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:18.186992    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:18.187192 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.187145    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:18.187258 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.187210    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.187195551 +0000 UTC m=+33.442040068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:18.194142 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:18.194114    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:18.194261 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:18.194125    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:18.194261 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.194220    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:18.194261 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:18.194231    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:18.194420 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.194336    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:18.194420 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.194403    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:18.288045 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:18.288015    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:18.288214 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.288159    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:18.288214 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.288174    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:18.288214 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.288183    2568 projected.go:196] Error preparing data for projected volume kube-api-access-7695w for pod openshift-network-diagnostics/network-check-target-pt4mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:18.288339 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.288245    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w podName:d94a8213-30da-4eef-9967-8afc5725b7db nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.288221381 +0000 UTC m=+33.543065898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7695w" (UniqueName: "kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w") pod "network-check-target-pt4mt" (UID: "d94a8213-30da-4eef-9967-8afc5725b7db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:18.389264 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:18.389234    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:18.389655 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.389377    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:18.389655 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:18.389436    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret podName:710b36be-68a9-4ed5-84c6-30d5e3dc6801 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.389423106 +0000 UTC m=+33.644267623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret") pod "global-pull-secret-syncer-x6kbk" (UID: "710b36be-68a9-4ed5-84c6-30d5e3dc6801") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:19.370659 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:19.370634    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"5b2c1bf798236a4be5f7a3a6f5a31c13005f1b92403ee838e1d8ee74b1ab45a3"}
Oct 30 19:41:20.194484 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.194285    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:20.195168 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.194306    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:20.195168 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:20.194567    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:20.195168 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.194326    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:20.195168 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:20.194671    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:20.195168 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:20.194715    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:20.375840 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.375806    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"d4531a053dbdcd5b67470c14aa6852c08ad270022f72088a68e889d99ac12008"}
Oct 30 19:41:20.375999 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.375908    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"00ad444390138715ca29795d4a2c043ad5f9d2966c0a3fccc114a31fc5eafab6"}
Oct 30 19:41:20.375999 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.375948    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"6c4409ed7b3a44be92fee92a8f9592e1f473476fcf76704f8b65b36a949adb26"}
Oct 30 19:41:20.375999 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.375985    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"54e1ddb06322cc5fb1a14f6cd41a02fae46a347ce2e5cff830d9dd19ac9627b0"}
Oct 30 19:41:20.376135 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.376013    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"53411383d3c537e0ff480cb8fb4fb90149c06c56fd9df3fad1286dfa941197ce"}
Oct 30 19:41:20.378413 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.378385    2568 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="859bd57a435ede42dc26946348a619c9a94e87086f9a2a8a3861eb115b6a2d25" exitCode=0
Oct 30 19:41:20.378512 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:20.378431    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerDied","Data":"859bd57a435ede42dc26946348a619c9a94e87086f9a2a8a3861eb115b6a2d25"}
Oct 30 19:41:21.383075 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:21.383039    2568 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="ee9e449e528cac299c509802af5fd7f331fc6cd8cb3382caea649cd34c89af75" exitCode=0
Oct 30 19:41:21.383485 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:21.383095    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerDied","Data":"ee9e449e528cac299c509802af5fd7f331fc6cd8cb3382caea649cd34c89af75"}
Oct 30 19:41:22.193566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:22.193544    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:22.193705 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:22.193575    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:22.193705 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:22.193639    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:22.193705 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:22.193677    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:22.193825 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:22.193759    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:22.193874 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:22.193856    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:22.388805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:22.388759    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lngvb" event={"ID":"0f76c795-db38-4b17-b2d5-c3c3d9905edc","Type":"ContainerStarted","Data":"8cbb8ff7b0a91335f17df27742d735dc9dea23b8e59a6f6774c3527dd60134a4"}
Oct 30 19:41:22.391804 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:22.391783    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"9d780e658ec7d9739a0be0ec183bc551e6aaca1f953c09fb3921823d3bff8d6a"}
Oct 30 19:41:22.520867 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:22.520784    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lngvb" podStartSLOduration=6.923863394 podStartE2EDuration="20.520756335s" totalImagesPullingTime="13.596892941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:22.520624484 +0000 UTC m=+21.775469024" watchObservedRunningTime="2026-10-30 19:41:22.520756335 +0000 UTC m=+21.775600875"
Oct 30 19:41:24.193636 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.193437    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:24.194026 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.193458    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:24.194026 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:24.193719    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:24.194026 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.193467    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:24.194026 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:24.193763    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:24.194026 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:24.193859    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:24.398270 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.398237    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" event={"ID":"4d555bfc-7445-4727-a110-ce66dae3ee49","Type":"ContainerStarted","Data":"c8e501d694515fe69ba88b8e4a4d4921c3dc95eec5cb02e88b242baa7f190e62"}
Oct 30 19:41:24.398489 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.398470    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:24.413483 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.413462    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:24.831358 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:24.831306    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7" podStartSLOduration=6.382557708 podStartE2EDuration="22.831288216s" totalImagesPullingTime="16.448730508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:24.695922286 +0000 UTC m=+23.950766826" watchObservedRunningTime="2026-10-30 19:41:24.831288216 +0000 UTC m=+24.086132756"
Oct 30 19:41:25.400384 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:25.400356    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:25.400856 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:25.400503    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:25.415898 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:25.415876    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:26.194327 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.194294    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:26.194453 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.194365    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:26.194501 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:26.194462    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:26.194536 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.194518    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:26.194636 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:26.194613    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:26.194703 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:26.194683    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:26.580144 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.580067    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tfswf"]
Oct 30 19:41:26.580516 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.580177    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:26.580516 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:26.580272    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:26.581923 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.581898    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pt4mt"]
Oct 30 19:41:26.582045 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.581976    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:26.582117 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:26.582043    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:26.586592 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.586567    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6kbk"]
Oct 30 19:41:26.586695 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:26.586653    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:26.586749 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:26.586731    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:28.194545 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:28.194516    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:28.195194 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:28.195166    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:28.195315 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:28.194644    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:28.195506 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:28.195490    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:28.195578 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:28.194583    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:28.195760 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:28.195744    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:30.193624 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:30.193600    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:30.193624 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:30.193600    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:30.194196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:30.193712    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:30.194196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:30.193720    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:30.194196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:30.193849    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:30.194196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:30.193917    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:32.194420 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.194259    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:32.194843 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.194298    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:32.194843 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.194499    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt4mt" podUID="d94a8213-30da-4eef-9967-8afc5725b7db"
Oct 30 19:41:32.194843 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.194298    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:32.194843 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.194618    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6kbk" podUID="710b36be-68a9-4ed5-84c6-30d5e3dc6801"
Oct 30 19:41:32.194843 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.194694    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfswf" podUID="eb6f0ca7-6e21-4447-a036-2c91f24e885f"
Oct 30 19:41:32.569752 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.569684    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:41:32.569897 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.569813    2568 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:41:32.662454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.662425    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-swx4r"]
Oct 30 19:41:32.666943 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.666926    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:32.670203 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.670179    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8wclq"]
Oct 30 19:41:32.672708 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.672691    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txjbg"]
Oct 30 19:41:32.672819 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.672806    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:32.675154 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.675138    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:32.687550 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.687525    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b45499cc5-52dbl"]
Oct 30 19:41:32.690285 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.690266    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-csrgl"]
Oct 30 19:41:32.690383 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.690368    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.693805 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.693786    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"]
Oct 30 19:41:32.693884 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.693845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:32.696594 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.696576    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"]
Oct 30 19:41:32.696820 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.696799    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.699482 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.699461    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"]
Oct 30 19:41:32.699557 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.699545    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.702549 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.702530    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"]
Oct 30 19:41:32.702752 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.702734    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.709407 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.708241    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"]
Oct 30 19:41:32.709407 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.708610    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.711763 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.711743    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"]
Oct 30 19:41:32.711882 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.711871    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:32.714850 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.714834    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-569cc6f5cf-pw4pl"]
Oct 30 19:41:32.714974 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.714957    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:32.718299 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.718281    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"]
Oct 30 19:41:32.718437 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.718419    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:32.721728 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.721711    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d57c9fd74-7lvhr"]
Oct 30 19:41:32.721832 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.721813    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:32.724687 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.724670    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-t9qb9"]
Oct 30 19:41:32.724812 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.724799    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.727342 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.727327    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9n45f"]
Oct 30 19:41:32.727467 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.727442    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.730340 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.730323    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9wckj"]
Oct 30 19:41:32.730424 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.730412    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:32.730955 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.730940    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.731036 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.730993    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.734092 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.734076    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-swx4r"]
Oct 30 19:41:32.734170 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.734158    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:32.760932 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.760913    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txjbg"]
Oct 30 19:41:32.770086 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.770064    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8wclq"]
Oct 30 19:41:32.772036 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772011    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:41:32.772136 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772087    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:41:32.772136 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772089    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:41:32.772237 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772178    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:41:32.772237 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772231    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:41:32.772689 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772673    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:41:32.772844 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772827    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpdkk\""
Oct 30 19:41:32.772961 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.772906    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:41:32.773037 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.773023    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hjc6w\""
Oct 30 19:41:32.781282 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.781263    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-csrgl"]
Oct 30 19:41:32.789520 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.789494    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"]
Oct 30 19:41:32.790567 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.790546    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"]
Oct 30 19:41:32.791278 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.791256    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b45499cc5-52dbl"]
Oct 30 19:41:32.795396 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795374    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31ae3d8c-faa0-468e-a557-d9f870fad555-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.795497 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795400    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:32.795497 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795418    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.795497 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795439    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0070d9d-efde-4889-87e9-39b47aefbe58-serving-cert\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.795497 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795484    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-trusted-ca\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.795695 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795555    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcw28\" (UniqueName: \"kubernetes.io/projected/19670c03-8928-4c59-b153-b51d3a77f63e-kube-api-access-zcw28\") pod \"kas-connection-checker-685895c7c9-8wclq\" (UID: \"19670c03-8928-4c59-b153-b51d3a77f63e\") " pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:32.795695 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795588    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.795695 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795664    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0070d9d-efde-4889-87e9-39b47aefbe58-config\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.795695 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795689    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.795915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795707    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-config\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.795915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795722    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.795915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795738    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.795915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795837    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdjv\" (UniqueName: \"kubernetes.io/projected/7ae66fed-db02-4f76-99ec-a80127e2a76a-kube-api-access-wmdjv\") pod \"kas-connection-checker-685895c7c9-txjbg\" (UID: \"7ae66fed-db02-4f76-99ec-a80127e2a76a\") " pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:32.795915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795856    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5td9\" (UniqueName: \"kubernetes.io/projected/9e23665b-8521-44d0-b403-ff1fefae7d74-kube-api-access-j5td9\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.795915 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795898    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31ae3d8c-faa0-468e-a557-d9f870fad555-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795939    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rfv\" (UniqueName: \"kubernetes.io/projected/31ae3d8c-faa0-468e-a557-d9f870fad555-kube-api-access-25rfv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795976    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.795994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e807294-3926-40b1-9203-9cbaf6f2ea40-nginx-conf\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796011    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e23665b-8521-44d0-b403-ff1fefae7d74-serving-cert\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796044    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796145    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5s4\" (UniqueName: \"kubernetes.io/projected/a0070d9d-efde-4889-87e9-39b47aefbe58-kube-api-access-ft5s4\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.796175 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796171    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqr7\" (UniqueName: \"kubernetes.io/projected/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-kube-api-access-nwqr7\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.796533 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796201    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9h95\" (UniqueName: \"kubernetes.io/projected/2cc8f2c8-4792-4fbf-b958-655692c689c7-kube-api-access-b9h95\") pod \"kas-connection-checker-685895c7c9-swx4r\" (UID: \"2cc8f2c8-4792-4fbf-b958-655692c689c7\") " pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:32.796533 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796240    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.796533 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.796257    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.812105 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.812082    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"]
Oct 30 19:41:32.820293 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.820239    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"]
Oct 30 19:41:32.821450 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.821396    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d57c9fd74-7lvhr"]
Oct 30 19:41:32.823570 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.823551    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"]
Oct 30 19:41:32.828479 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.828460    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-569cc6f5cf-pw4pl"]
Oct 30 19:41:32.837268 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.837250    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.837662 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.837648    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:41:32.837783 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.837760    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.837952 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.837762    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.837952 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.837797    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.839245 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.839223    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.839321 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.839255    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:41:32.839360 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.839283    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:41:32.839889 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.839859    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:41:32.840201 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.840179    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.840439 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.840234    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:41:32.841030 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841002    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.841185 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841162    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.841244 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841221    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.841488 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841472    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.841650 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841637    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.841704 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841678    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.841753 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.841738    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:41:32.842088 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.842071    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.842149 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.842139    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.842198 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.842148    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.842276 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.842259    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.887286 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.887249    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:41:32.890022 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.890000    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:32.891591 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891568    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891634    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891675    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891694    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891702    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891715    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891748    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:41:32.891761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891750    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891764    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891823    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891846    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891863    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891863    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891893    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pxncz\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.891916    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:41:32.892191 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.892047    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgpv7\""
Oct 30 19:41:32.892547 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.892529    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:41:32.892632 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.892608    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:41:32.893359 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.893342    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"]
Oct 30 19:41:32.894047 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.894033    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9wckj"]
Oct 30 19:41:32.894660 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.894644    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"]
Oct 30 19:41:32.895756 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.895742    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-t9qb9"]
Oct 30 19:41:32.897320 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897301    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrgh\" (UniqueName: \"kubernetes.io/projected/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-kube-api-access-fxrgh\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:32.897366 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.897398 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.897429 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897410    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmj2j\" (UniqueName: \"kubernetes.io/projected/4fe3fb9e-3883-4afe-ad11-64cd33045396-kube-api-access-lmj2j\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:32.897503 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0070d9d-efde-4889-87e9-39b47aefbe58-config\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.897554 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897537    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.897589 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897570    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-config\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.897619 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.897657 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.897657 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.897638    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:32.897657 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.897650    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:32.897657 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-default-certificate\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:32.897821 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.897701    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.397684224 +0000 UTC m=+32.652528754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:32.897821 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897788    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5td9\" (UniqueName: \"kubernetes.io/projected/9e23665b-8521-44d0-b403-ff1fefae7d74-kube-api-access-j5td9\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.897821 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897809    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:32.897971 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-stats-auth\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:32.897971 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897841    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5cj\" (UniqueName: \"kubernetes.io/projected/0aefa597-5836-435a-9bc8-b54a2908db1c-kube-api-access-ms5cj\") pod \"volume-data-source-validator-58c9f5f46d-768l8\" (UID: \"0aefa597-5836-435a-9bc8-b54a2908db1c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:32.897971 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.897855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:32.898119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898064    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.898119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898083    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rfv\" (UniqueName: \"kubernetes.io/projected/31ae3d8c-faa0-468e-a557-d9f870fad555-kube-api-access-25rfv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.898119 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898089    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0070d9d-efde-4889-87e9-39b47aefbe58-config\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.898222 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898116    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.898222 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31ae3d8c-faa0-468e-a557-d9f870fad555-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.898222 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898178    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.898222 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898207    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de6c843-5c4d-4de8-99ff-950dd189b6bd-tmp-dir\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:32.898413 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898236    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e807294-3926-40b1-9203-9cbaf6f2ea40-nginx-conf\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:32.898413 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898263    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.898413 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898303    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:32.898413 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898365    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-config\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.898555 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898435    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5s4\" (UniqueName: \"kubernetes.io/projected/a0070d9d-efde-4889-87e9-39b47aefbe58-kube-api-access-ft5s4\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.898555 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.898555 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898472    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.898555 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31ae3d8c-faa0-468e-a557-d9f870fad555-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.898555 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898536    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898565    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.898583    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-trusted-ca\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898628    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcw28\" (UniqueName: \"kubernetes.io/projected/19670c03-8928-4c59-b153-b51d3a77f63e-kube-api-access-zcw28\") pod \"kas-connection-checker-685895c7c9-8wclq\" (UID: \"19670c03-8928-4c59-b153-b51d3a77f63e\") " pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.898649    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.39862183 +0000 UTC m=+32.653466347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898684    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898727    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866th\" (UniqueName: \"kubernetes.io/projected/5de6c843-5c4d-4de8-99ff-950dd189b6bd-kube-api-access-866th\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:32.898801 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szpb\" (UniqueName: \"kubernetes.io/projected/83293a26-814d-4b38-bf8e-10b28947c6c2-kube-api-access-9szpb\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898809    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83293a26-814d-4b38-bf8e-10b28947c6c2-serving-cert\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898836    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898868    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnwh\" (UniqueName: \"kubernetes.io/projected/5b117693-4843-4840-a6f1-5567a04226e1-kube-api-access-2lnwh\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898936    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e807294-3926-40b1-9203-9cbaf6f2ea40-nginx-conf\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.898969    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-snapshots\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899026    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.899184 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdjv\" (UniqueName: \"kubernetes.io/projected/7ae66fed-db02-4f76-99ec-a80127e2a76a-kube-api-access-wmdjv\") pod \"kas-connection-checker-685895c7c9-txjbg\" (UID: \"7ae66fed-db02-4f76-99ec-a80127e2a76a\") " pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:32.899566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.899566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899384    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31ae3d8c-faa0-468e-a557-d9f870fad555-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.899566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-tmp\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.899566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899524    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-service-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:32.899566 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899566    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe3fb9e-3883-4afe-ad11-64cd33045396-cert\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899606    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-trusted-ca\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5de6c843-5c4d-4de8-99ff-950dd189b6bd-config-volume\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899638    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899673    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5m4\" (UniqueName: \"kubernetes.io/projected/87025e83-063c-4c2b-9ba2-e551bd098d29-kube-api-access-nz5m4\") pod \"network-check-source-6f78d59f99-69s2p\" (UID: \"87025e83-063c-4c2b-9ba2-e551bd098d29\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899714    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e23665b-8521-44d0-b403-ff1fefae7d74-serving-cert\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899746    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.899795 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899764    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqr7\" (UniqueName: \"kubernetes.io/projected/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-kube-api-access-nwqr7\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.900196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899840    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h95\" (UniqueName: \"kubernetes.io/projected/2cc8f2c8-4792-4fbf-b958-655692c689c7-kube-api-access-b9h95\") pod \"kas-connection-checker-685895c7c9-swx4r\" (UID: \"2cc8f2c8-4792-4fbf-b958-655692c689c7\") " pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:32.900196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899847    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.900196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899875    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:32.900196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:32.900196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.899939    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0070d9d-efde-4889-87e9-39b47aefbe58-serving-cert\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.900196 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.900130    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.900480 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.900215    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:32.900480 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:32.900265    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.400251632 +0000 UTC m=+32.655096164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:32.903055 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.902737    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9n45f"]
Oct 30 19:41:32.903264 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.903246    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0070d9d-efde-4889-87e9-39b47aefbe58-serving-cert\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.903327 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.903293    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e23665b-8521-44d0-b403-ff1fefae7d74-serving-cert\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.903365 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.903337    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.904370 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.904346    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.906260 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.906232    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31ae3d8c-faa0-468e-a557-d9f870fad555-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.963258 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.963229    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.968937 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.968904    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdjv\" (UniqueName: \"kubernetes.io/projected/7ae66fed-db02-4f76-99ec-a80127e2a76a-kube-api-access-wmdjv\") pod \"kas-connection-checker-685895c7c9-txjbg\" (UID: \"7ae66fed-db02-4f76-99ec-a80127e2a76a\") " pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:32.969107 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.969089    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcw28\" (UniqueName: \"kubernetes.io/projected/19670c03-8928-4c59-b153-b51d3a77f63e-kube-api-access-zcw28\") pod \"kas-connection-checker-685895c7c9-8wclq\" (UID: \"19670c03-8928-4c59-b153-b51d3a77f63e\") " pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:32.969185 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.969165    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h95\" (UniqueName: \"kubernetes.io/projected/2cc8f2c8-4792-4fbf-b958-655692c689c7-kube-api-access-b9h95\") pod \"kas-connection-checker-685895c7c9-swx4r\" (UID: \"2cc8f2c8-4792-4fbf-b958-655692c689c7\") " pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:32.976437 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.976412    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:32.977330 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.977314    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqr7\" (UniqueName: \"kubernetes.io/projected/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-kube-api-access-nwqr7\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:32.978109 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.978089    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:32.981642 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.981621    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rfv\" (UniqueName: \"kubernetes.io/projected/31ae3d8c-faa0-468e-a557-d9f870fad555-kube-api-access-25rfv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:32.981756 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.981690    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5s4\" (UniqueName: \"kubernetes.io/projected/a0070d9d-efde-4889-87e9-39b47aefbe58-kube-api-access-ft5s4\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:32.982347 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.982329    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5td9\" (UniqueName: \"kubernetes.io/projected/9e23665b-8521-44d0-b403-ff1fefae7d74-kube-api-access-j5td9\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:32.985177 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.985161    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:32.991308 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:32.991289    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:33.000441 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000419    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de6c843-5c4d-4de8-99ff-950dd189b6bd-tmp-dir\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.000551 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.000551 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000464    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.000551 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000496    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:33.000551 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000525    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866th\" (UniqueName: \"kubernetes.io/projected/5de6c843-5c4d-4de8-99ff-950dd189b6bd-kube-api-access-866th\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.000551 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szpb\" (UniqueName: \"kubernetes.io/projected/83293a26-814d-4b38-bf8e-10b28947c6c2-kube-api-access-9szpb\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.000557    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83293a26-814d-4b38-bf8e-10b28947c6c2-serving-cert\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.000610    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.500592167 +0000 UTC m=+32.755436721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000633    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnwh\" (UniqueName: \"kubernetes.io/projected/5b117693-4843-4840-a6f1-5567a04226e1-kube-api-access-2lnwh\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000702    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-snapshots\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.000785 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000753    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000792    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-tmp\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000816    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-service-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000840    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe3fb9e-3883-4afe-ad11-64cd33045396-cert\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000871    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5de6c843-5c4d-4de8-99ff-950dd189b6bd-config-volume\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000878    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de6c843-5c4d-4de8-99ff-950dd189b6bd-tmp-dir\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000900    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000926    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5m4\" (UniqueName: \"kubernetes.io/projected/87025e83-063c-4c2b-9ba2-e551bd098d29-kube-api-access-nz5m4\") pod \"network-check-source-6f78d59f99-69s2p\" (UID: \"87025e83-063c-4c2b-9ba2-e551bd098d29\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000957    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.000989    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001030    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrgh\" (UniqueName: \"kubernetes.io/projected/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-kube-api-access-fxrgh\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001058    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.001101 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001083    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001113    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmj2j\" (UniqueName: \"kubernetes.io/projected/4fe3fb9e-3883-4afe-ad11-64cd33045396-kube-api-access-lmj2j\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001165    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001191    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-default-certificate\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001237    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001255    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-tmp\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-stats-auth\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001267    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001293    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5cj\" (UniqueName: \"kubernetes.io/projected/0aefa597-5836-435a-9bc8-b54a2908db1c-kube-api-access-ms5cj\") pod \"volume-data-source-validator-58c9f5f46d-768l8\" (UID: \"0aefa597-5836-435a-9bc8-b54a2908db1c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001322    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001364    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.001437    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.001482    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.501467347 +0000 UTC m=+32.756311866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.001498    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.50148926 +0000 UTC m=+32.756333778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001507    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5de6c843-5c4d-4de8-99ff-950dd189b6bd-config-volume\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.001525    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:33.001686 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.001541    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.001589    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.501570304 +0000 UTC m=+32.756414835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.001887    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-snapshots\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.002105    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-service-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.002127    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.002792    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.002849    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.502834873 +0000 UTC m=+32.757679413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:33.003042 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.002858    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.003282 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.003261    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83293a26-814d-4b38-bf8e-10b28947c6c2-serving-cert\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.003320 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.003280    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.004116 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.004087    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.004305 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.004285    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-default-certificate\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.004497 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.004474    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe3fb9e-3883-4afe-ad11-64cd33045396-cert\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:33.004630 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.004616    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.005197 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.005182    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-stats-auth\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.016237 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.016133    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:33.023946 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.023914    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:33.029032 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.029012    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:33.092491 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.092399    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.093594 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.093562    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5cj\" (UniqueName: \"kubernetes.io/projected/0aefa597-5836-435a-9bc8-b54a2908db1c-kube-api-access-ms5cj\") pod \"volume-data-source-validator-58c9f5f46d-768l8\" (UID: \"0aefa597-5836-435a-9bc8-b54a2908db1c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:33.094998 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.094974    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5m4\" (UniqueName: \"kubernetes.io/projected/87025e83-063c-4c2b-9ba2-e551bd098d29-kube-api-access-nz5m4\") pod \"network-check-source-6f78d59f99-69s2p\" (UID: \"87025e83-063c-4c2b-9ba2-e551bd098d29\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:33.095127 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.095104    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szpb\" (UniqueName: \"kubernetes.io/projected/83293a26-814d-4b38-bf8e-10b28947c6c2-kube-api-access-9szpb\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.095538 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.095493    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.097826 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.097797    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmj2j\" (UniqueName: \"kubernetes.io/projected/4fe3fb9e-3883-4afe-ad11-64cd33045396-kube-api-access-lmj2j\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:33.101817 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.101326    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866th\" (UniqueName: \"kubernetes.io/projected/5de6c843-5c4d-4de8-99ff-950dd189b6bd-kube-api-access-866th\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.101817 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.101429    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrgh\" (UniqueName: \"kubernetes.io/projected/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-kube-api-access-fxrgh\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:33.102114 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.102005    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnwh\" (UniqueName: \"kubernetes.io/projected/5b117693-4843-4840-a6f1-5567a04226e1-kube-api-access-2lnwh\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.342246 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.342215    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:33.359316 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.359250    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:33.373529 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.373505    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:33.382687 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.382662    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:33.404999 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.404974    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:33.405110 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.405080    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:33.405144 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405129    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:33.405184 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405144    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:33.405218 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.405179    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:33.405218 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405200    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.40518213 +0000 UTC m=+33.660026662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:33.405279 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405243    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:33.405310 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405279    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.405268766 +0000 UTC m=+33.660113283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:33.405345 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405329    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:33.405376 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.405359    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.405350128 +0000 UTC m=+33.660194662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:33.434031 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.434003    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"]
Oct 30 19:41:33.439369 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.439288    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ae3d8c_faa0_468e_a557_d9f870fad555.slice/crio-3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd WatchSource:0}: Error finding container 3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd: Status 404 returned error can't find the container with id 3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd
Oct 30 19:41:33.440380 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.440351    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txjbg"]
Oct 30 19:41:33.450401 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.444661    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"]
Oct 30 19:41:33.464410 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.464276    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0070d9d_efde_4889_87e9_39b47aefbe58.slice/crio-852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c WatchSource:0}: Error finding container 852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c: Status 404 returned error can't find the container with id 852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c
Oct 30 19:41:33.486648 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.486621    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"]
Oct 30 19:41:33.491901 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.491846    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e23665b_8521_44d0_b403_ff1fefae7d74.slice/crio-1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a WatchSource:0}: Error finding container 1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a: Status 404 returned error can't find the container with id 1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a
Oct 30 19:41:33.505236 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.505165    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8wclq"]
Oct 30 19:41:33.506651 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.506622    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.506786 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.506673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:33.506786 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.506731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:33.506786 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.506751    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:33.506951 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.506819    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:33.507007 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.506968    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:33.507052 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507025    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.507007387 +0000 UTC m=+33.761851925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:33.507111 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507096    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.507086142 +0000 UTC m=+33.761930676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:33.507167 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507159    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:33.507219 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507170    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:33.507219 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507201    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.507191981 +0000 UTC m=+33.762036515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:33.507324 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507253    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:33.507324 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507281    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.507271631 +0000 UTC m=+33.762116165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:33.507426 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507333    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:33.507426 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:33.507361    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:34.507352334 +0000 UTC m=+33.762196865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:33.516088 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.514721    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19670c03_8928_4c59_b153_b51d3a77f63e.slice/crio-f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e WatchSource:0}: Error finding container f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e: Status 404 returned error can't find the container with id f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e
Oct 30 19:41:33.516088 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.515115    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-swx4r"]
Oct 30 19:41:33.518218 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.517916    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc8f2c8_4792_4fbf_b958_655692c689c7.slice/crio-e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe WatchSource:0}: Error finding container e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe: Status 404 returned error can't find the container with id e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe
Oct 30 19:41:33.740367 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.740332    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"]
Oct 30 19:41:33.742823 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.742791    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aefa597_5836_435a_9bc8_b54a2908db1c.slice/crio-434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c WatchSource:0}: Error finding container 434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c: Status 404 returned error can't find the container with id 434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c
Oct 30 19:41:33.779798 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.779759    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"]
Oct 30 19:41:33.781268 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.781236    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87025e83_063c_4c2b_9ba2_e551bd098d29.slice/crio-0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2 WatchSource:0}: Error finding container 0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2: Status 404 returned error can't find the container with id 0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2
Oct 30 19:41:33.830248 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.830209    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-t9qb9"]
Oct 30 19:41:33.832810 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.832783    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83293a26_814d_4b38_bf8e_10b28947c6c2.slice/crio-d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5 WatchSource:0}: Error finding container d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5: Status 404 returned error can't find the container with id d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5
Oct 30 19:41:33.864105 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:33.864083    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9n45f"]
Oct 30 19:41:33.866140 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:33.866098    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe3fb9e_3883_4afe_ad11_64cd33045396.slice/crio-6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2 WatchSource:0}: Error finding container 6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2: Status 404 returned error can't find the container with id 6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2
Oct 30 19:41:34.193687 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.193660    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:34.193850 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.193695    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:34.193850 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.193801    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:34.212868 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.212848    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:34.260910 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.260887    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:41:34.261046 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.260920    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\""
Oct 30 19:41:34.261046 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.260920    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\""
Oct 30 19:41:34.261046 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.260951    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\""
Oct 30 19:41:34.261046 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.261031    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:41:34.263117 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.263100    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:41:34.263208 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.263176    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:42:06.263152308 +0000 UTC m=+65.517996826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : secret "metrics-daemon-secret" not found
Oct 30 19:41:34.313553 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.313525    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:34.315911 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.315879    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:34.414235 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.414208    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.414296    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414361    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414381    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.414374    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414441    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.41442104 +0000 UTC m=+35.669265559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414456    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414509    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.414492637 +0000 UTC m=+35.669337154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:34.414658 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.414568    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:34.414966 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414672    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:34.414966 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.414705    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.414697086 +0000 UTC m=+35.669541603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:34.416646 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.416603    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:34.420171 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.420146    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8wclq" event={"ID":"19670c03-8928-4c59-b153-b51d3a77f63e","Type":"ContainerStarted","Data":"b0564a0db91c2442a6bb808883942a670d063a0104770ba907f333ea48c6d957"}
Oct 30 19:41:34.420289 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.420230    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8wclq" event={"ID":"19670c03-8928-4c59-b153-b51d3a77f63e","Type":"ContainerStarted","Data":"f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e"}
Oct 30 19:41:34.421496 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.421473    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9n45f" event={"ID":"4fe3fb9e-3883-4afe-ad11-64cd33045396","Type":"ContainerStarted","Data":"6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2"}
Oct 30 19:41:34.422738 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.422716    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerStarted","Data":"1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a"}
Oct 30 19:41:34.424080 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.424052    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-txjbg" event={"ID":"7ae66fed-db02-4f76-99ec-a80127e2a76a","Type":"ContainerStarted","Data":"25683995e2ac8a5f4097665274a607b482eba306f7b964a8a20a83375e622198"}
Oct 30 19:41:34.424173 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.424098    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-txjbg" event={"ID":"7ae66fed-db02-4f76-99ec-a80127e2a76a","Type":"ContainerStarted","Data":"5c7b219c376a83723daa231618c253ed4344742dc57ce49c6005efd0aeae4ca7"}
Oct 30 19:41:34.425188 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.425167    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c" event={"ID":"31ae3d8c-faa0-468e-a557-d9f870fad555","Type":"ContainerStarted","Data":"3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd"}
Oct 30 19:41:34.426477 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.426458    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-swx4r" event={"ID":"2cc8f2c8-4792-4fbf-b958-655692c689c7","Type":"ContainerStarted","Data":"76fa9ad1a7216734e3397d210724c3c9d4ba32cdf433ffbd287afe9b2d81cbb9"}
Oct 30 19:41:34.426551 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.426492    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-swx4r" event={"ID":"2cc8f2c8-4792-4fbf-b958-655692c689c7","Type":"ContainerStarted","Data":"e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe"}
Oct 30 19:41:34.427521 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.427501    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t9qb9" event={"ID":"83293a26-814d-4b38-bf8e-10b28947c6c2","Type":"ContainerStarted","Data":"d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5"}
Oct 30 19:41:34.428416 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.428400    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p" event={"ID":"87025e83-063c-4c2b-9ba2-e551bd098d29","Type":"ContainerStarted","Data":"0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2"}
Oct 30 19:41:34.429328 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.429305    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8" event={"ID":"0aefa597-5836-435a-9bc8-b54a2908db1c","Type":"ContainerStarted","Data":"434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c"}
Oct 30 19:41:34.430217 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.430201    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv" event={"ID":"a0070d9d-efde-4889-87e9-39b47aefbe58","Type":"ContainerStarted","Data":"852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c"}
Oct 30 19:41:34.515589 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.515506    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:34.515589 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.515550    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.515608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.515635    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.515667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515710    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.515671983 +0000 UTC m=+35.770516523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515745    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515757    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515802    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:34.515813 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515812    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.515796458 +0000 UTC m=+35.770640995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:34.516196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515718    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:34.516196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515834    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.515824677 +0000 UTC m=+35.770669197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:34.516196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515841    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:34.516196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515861    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.51584738 +0000 UTC m=+35.770691911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:34.516196 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:34.515880    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.515870975 +0000 UTC m=+35.770715496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:34.539691 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.539663    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:34.563947 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.563918    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:34.956815 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.956762    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6kbk"]
Oct 30 19:41:34.990835 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:34.990812    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pt4mt"]
Oct 30 19:41:34.994365 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:34.994331    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94a8213_30da_4eef_9967_8afc5725b7db.slice/crio-2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6 WatchSource:0}: Error finding container 2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6: Status 404 returned error can't find the container with id 2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6
Oct 30 19:41:35.220028 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:35.219922    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8wclq" podStartSLOduration=337.219902579 podStartE2EDuration="5m37.219902579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:35.088951456 +0000 UTC m=+34.343796022" watchObservedRunningTime="2026-10-30 19:41:35.219902579 +0000 UTC m=+34.474747122"
Oct 30 19:41:35.343098 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:35.343039    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-swx4r" podStartSLOduration=337.343019063 podStartE2EDuration="5m37.343019063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:35.220838973 +0000 UTC m=+34.475683532" watchObservedRunningTime="2026-10-30 19:41:35.343019063 +0000 UTC m=+34.597863602"
Oct 30 19:41:35.343295 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:35.343142    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-txjbg" podStartSLOduration=337.343137336 podStartE2EDuration="5m37.343137336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:35.34128149 +0000 UTC m=+34.596126043" watchObservedRunningTime="2026-10-30 19:41:35.343137336 +0000 UTC m=+34.597981877"
Oct 30 19:41:35.436653 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:35.436619    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pt4mt" event={"ID":"d94a8213-30da-4eef-9967-8afc5725b7db","Type":"ContainerStarted","Data":"2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6"}
Oct 30 19:41:35.442492 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:35.442463    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6kbk" event={"ID":"710b36be-68a9-4ed5-84c6-30d5e3dc6801","Type":"ContainerStarted","Data":"8b8465855c918b693eb982a133e7a9a15f91fe05a0490dbccd060c16e3ca1090"}
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.437016    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.437088    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437156    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.437196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437222    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.437203411 +0000 UTC m=+39.692047947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437284    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437333    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.437318491 +0000 UTC m=+39.692163011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437400    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437411    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:36.437464 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.437443    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.437432412 +0000 UTC m=+39.692276931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:36.538051 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.538011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:36.538230 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.538102    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:36.538230 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.538124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:36.538230 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.538179    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:36.538230 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:36.538201    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:36.538436 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538310    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:36.538436 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538356    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.538344824 +0000 UTC m=+39.793189354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:36.538539 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538480    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:36.538539 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538502    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.53849503 +0000 UTC m=+39.793339560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:36.538642 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538546    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.538539964 +0000 UTC m=+39.793384493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:36.538642 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538585    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:36.538642 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538593    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:36.538642 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538611    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.538605814 +0000 UTC m=+39.793450344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:36.538642 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538642    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:36.538900 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:36.538658    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.538653642 +0000 UTC m=+39.793498171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:37.452870 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:37.452261    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8" event={"ID":"0aefa597-5836-435a-9bc8-b54a2908db1c","Type":"ContainerStarted","Data":"e30c883404cc5720965abd5af8bec8f6dc4b1971cc4517b415e78cb9d3882836"}
Oct 30 19:41:37.582922 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:37.582857    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8" podStartSLOduration=308.472501644 podStartE2EDuration="5m11.582838607s" totalImagesPullingTime="3.110336963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:37.582022058 +0000 UTC m=+36.836866609" watchObservedRunningTime="2026-10-30 19:41:37.582838607 +0000 UTC m=+36.837683147"
Oct 30 19:41:39.460169 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:39.460132    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c" event={"ID":"31ae3d8c-faa0-468e-a557-d9f870fad555","Type":"ContainerStarted","Data":"d645aeff79c60a051076a93950491b26a79bbd8cba5994c0a79ed3f5d0ef6ff2"}
Oct 30 19:41:39.462147 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:39.462122    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/0.log"
Oct 30 19:41:39.462281 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:39.462167    2568 generic.go:356] "Generic (PLEG): container finished" podID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerID="779841cdd4c33ca59fbd1e7278e1e92ecb9f16707beaa0b523c49a9dff76f5f9" exitCode=255
Oct 30 19:41:39.462281 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:39.462199    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerDied","Data":"779841cdd4c33ca59fbd1e7278e1e92ecb9f16707beaa0b523c49a9dff76f5f9"}
Oct 30 19:41:39.462485 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:39.462469    2568 scope.go:122] "RemoveContainer" containerID="779841cdd4c33ca59fbd1e7278e1e92ecb9f16707beaa0b523c49a9dff76f5f9"
Oct 30 19:41:39.586932 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:39.586872    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c" podStartSLOduration=308.272570642 podStartE2EDuration="5m13.5868536s" totalImagesPullingTime="5.314282958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:39.585163567 +0000 UTC m=+38.840008101" watchObservedRunningTime="2026-10-30 19:41:39.5868536 +0000 UTC m=+38.841698141"
Oct 30 19:41:40.473587 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.473547    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv" event={"ID":"a0070d9d-efde-4889-87e9-39b47aefbe58","Type":"ContainerStarted","Data":"7b6e6adeb29f7437ca78d3df49e96b04290b5b3ec99e7779d8bf65ca9b1e35f8"}
Oct 30 19:41:40.476808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.475635    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9n45f" event={"ID":"4fe3fb9e-3883-4afe-ad11-64cd33045396","Type":"ContainerStarted","Data":"5125ae11cfd3441dc60e2059ac425e8291a57017d429750397413aae1439a176"}
Oct 30 19:41:40.479488 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.479352    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/1.log"
Oct 30 19:41:40.480808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.479823    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/0.log"
Oct 30 19:41:40.480808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.479855    2568 generic.go:356] "Generic (PLEG): container finished" podID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509" exitCode=255
Oct 30 19:41:40.480808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.479958    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerDied","Data":"b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509"}
Oct 30 19:41:40.480808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.479998    2568 scope.go:122] "RemoveContainer" containerID="779841cdd4c33ca59fbd1e7278e1e92ecb9f16707beaa0b523c49a9dff76f5f9"
Oct 30 19:41:40.480808 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.480418    2568 scope.go:122] "RemoveContainer" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509"
Oct 30 19:41:40.480808 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.480613    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:40.483853 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.482953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:40.483853 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.483033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:40.483853 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.483150    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:40.483853 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.483162    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:40.483853 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.483204    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.483189369 +0000 UTC m=+47.738033885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:40.483853 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.483651    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:40.484798 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.484572    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:40.484798 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.484635    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.484612438 +0000 UTC m=+47.739456961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:40.485657 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.484980    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:40.485657 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.485204    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.485191954 +0000 UTC m=+47.740036471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:40.490300 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.490275    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t9qb9" event={"ID":"83293a26-814d-4b38-bf8e-10b28947c6c2","Type":"ContainerStarted","Data":"d91726df0ef2995803cc01962f8e664712cc93933b2e508ae36048377eac691e"}
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.585235    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.585310    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.585445    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.585482    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.585577    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.585713    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.585691099 +0000 UTC m=+47.840535616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.585951    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.585968    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586021    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586021    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586060    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.586045723 +0000 UTC m=+47.840890257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586079    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.586069861 +0000 UTC m=+47.840914384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586106    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.586099326 +0000 UTC m=+47.840943878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586180    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:40.586454 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:40.586366    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.586348393 +0000 UTC m=+47.841192925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:40.598434 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.598375    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv" podStartSLOduration=308.373509483 podStartE2EDuration="5m14.598356078s" totalImagesPullingTime="6.224846595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.595730109 +0000 UTC m=+39.850574676" watchObservedRunningTime="2026-10-30 19:41:40.598356078 +0000 UTC m=+39.853200622"
Oct 30 19:41:40.761714 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.761335    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9n45f" podStartSLOduration=2.396999779 podStartE2EDuration="8.761314362s" totalImagesPullingTime="6.364314583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.731235347 +0000 UTC m=+39.986079911" watchObservedRunningTime="2026-10-30 19:41:40.761314362 +0000 UTC m=+40.016158904"
Oct 30 19:41:40.763646 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.762445    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"]
Oct 30 19:41:40.784400 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.784357    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:40.830681 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.830640    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"]
Oct 30 19:41:40.881518 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.881465    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-t9qb9" podStartSLOduration=309.046111286 podStartE2EDuration="5m14.881451662s" totalImagesPullingTime="5.835340376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.880842123 +0000 UTC m=+40.135686683" watchObservedRunningTime="2026-10-30 19:41:40.881451662 +0000 UTC m=+40.136296201"
Oct 30 19:41:40.888269 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.888237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtv7\" (UniqueName: \"kubernetes.io/projected/b1efd70d-88c5-4c06-bea1-b998190bb385-kube-api-access-fvtv7\") pod \"migrator-66df5d496b-kpp2d\" (UID: \"b1efd70d-88c5-4c06-bea1-b998190bb385\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:40.908404 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.908197    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:40.908404 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.908313    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v4lmq\""
Oct 30 19:41:40.908404 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.908375    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:41:40.989472 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:40.989439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtv7\" (UniqueName: \"kubernetes.io/projected/b1efd70d-88c5-4c06-bea1-b998190bb385-kube-api-access-fvtv7\") pod \"migrator-66df5d496b-kpp2d\" (UID: \"b1efd70d-88c5-4c06-bea1-b998190bb385\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:41.052579 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.052511    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtv7\" (UniqueName: \"kubernetes.io/projected/b1efd70d-88c5-4c06-bea1-b998190bb385-kube-api-access-fvtv7\") pod \"migrator-66df5d496b-kpp2d\" (UID: \"b1efd70d-88c5-4c06-bea1-b998190bb385\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:41.110864 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.110829    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:41.183761 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.183732    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nwlvj"]
Oct 30 19:41:41.208798 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.208539    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.261233 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.261203    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwlvj"]
Oct 30 19:41:41.281681 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.281648    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:41:41.281855 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.281723    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:41:41.282198 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.282181    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wfbg\""
Oct 30 19:41:41.294318 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.293877    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/292c4af8-ff18-45ab-96b2-bd05430793cf-crio-socket\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.294318 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.293920    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.294318 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.294004    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/292c4af8-ff18-45ab-96b2-bd05430793cf-data-volume\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.294318 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.294084    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.294318 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.294124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8kw7\" (UniqueName: \"kubernetes.io/projected/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-api-access-f8kw7\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395113 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395050    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/292c4af8-ff18-45ab-96b2-bd05430793cf-crio-socket\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395113 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395082    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395270 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.395163    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:41:41.395270 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395167    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/292c4af8-ff18-45ab-96b2-bd05430793cf-crio-socket\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395270 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.395212    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:41.895198876 +0000 UTC m=+41.150043412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : configmap "kube-rbac-proxy" not found
Oct 30 19:41:41.395270 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395236    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/292c4af8-ff18-45ab-96b2-bd05430793cf-data-volume\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395410 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395410 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kw7\" (UniqueName: \"kubernetes.io/projected/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-api-access-f8kw7\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.395480 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.395422    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:41.395480 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.395467    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:41.895453874 +0000 UTC m=+41.150298406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:41.395561 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.395545    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/292c4af8-ff18-45ab-96b2-bd05430793cf-data-volume\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.467528 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.467499    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kw7\" (UniqueName: \"kubernetes.io/projected/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-api-access-f8kw7\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.500026 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.499989    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p" event={"ID":"87025e83-063c-4c2b-9ba2-e551bd098d29","Type":"ContainerStarted","Data":"4307148d23ad554b6b33b4e9c441f6dc971f022fb9f4ef99f91f875996e8e6c5"}
Oct 30 19:41:41.501700 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.501677    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/1.log"
Oct 30 19:41:41.502134 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.502116    2568 scope.go:122] "RemoveContainer" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509"
Oct 30 19:41:41.502316 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.502295    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:41.526994 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.526966    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"]
Oct 30 19:41:41.529921 ip-10-0-133-27 kubenswrapper[2568]: W1030 19:41:41.529846    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1efd70d_88c5_4c06_bea1_b998190bb385.slice/crio-ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1 WatchSource:0}: Error finding container ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1: Status 404 returned error can't find the container with id ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1
Oct 30 19:41:41.745179 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.745125    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p" podStartSLOduration=302.405543163 podStartE2EDuration="5m9.745107524s" totalImagesPullingTime="7.339564361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:41.619549562 +0000 UTC m=+40.874394112" watchObservedRunningTime="2026-10-30 19:41:41.745107524 +0000 UTC m=+40.999952063"
Oct 30 19:41:41.903173 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.903141    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.903343 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:41.903268    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:41.903399 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.903270    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:41:41.903462 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.903450    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:42.903430985 +0000 UTC m=+42.158275515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : configmap "kube-rbac-proxy" not found
Oct 30 19:41:41.903523 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.903347    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:41.903571 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:41.903541    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:42.903523697 +0000 UTC m=+42.158368227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:42.506831 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:42.506792    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d" event={"ID":"b1efd70d-88c5-4c06-bea1-b998190bb385","Type":"ContainerStarted","Data":"ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1"}
Oct 30 19:41:42.508600 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:42.508574    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pt4mt" event={"ID":"d94a8213-30da-4eef-9967-8afc5725b7db","Type":"ContainerStarted","Data":"f6a797811d6499a38e0acad384e39ace91f884cc760eab74e68ceb37160d5ce1"}
Oct 30 19:41:42.508823 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:42.508801    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:42.635331 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:42.635274    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pt4mt" podStartSLOduration=33.888516262 podStartE2EDuration="40.63525976s" totalImagesPullingTime="6.746743498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:42.634689607 +0000 UTC m=+41.889534144" watchObservedRunningTime="2026-10-30 19:41:42.63525976 +0000 UTC m=+41.890104298"
Oct 30 19:41:42.912167 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:42.912069    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:42.912421 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:42.912186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:42.912421 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:42.912204    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:41:42.912421 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:42.912311    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:42.912421 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:42.912318    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:44.912296205 +0000 UTC m=+44.167140737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : configmap "kube-rbac-proxy" not found
Oct 30 19:41:42.912421 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:42.912354    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:44.91234161 +0000 UTC m=+44.167186127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:43.024635 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:43.024595    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:43.024822 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:43.024672    2568 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:43.025121 ip-10-0-133-27 kubenswrapper[2568]: I1030 19:41:43.025104    2568 scope.go:122] "RemoveContainer" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509"
Oct 30 19:41:43.025354 ip-10-0-133-27 kubenswrapper[2568]: E1030 19:41:43.025330    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:43.761430 ip-10-0-133-27 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:41:43.777219 ip-10-0-133-27 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:41:43.777389 ip-10-0-133-27 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:41:43.777442 ip-10-0-133-27 systemd[1]: kubelet.service: Consumed 3.020s CPU time.
Oct 30 19:41:43.790440 ip-10-0-133-27 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:41:43.892389 ip-10-0-133-27 kubenswrapper[4917]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:43.892389 ip-10-0-133-27 kubenswrapper[4917]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:41:43.892389 ip-10-0-133-27 kubenswrapper[4917]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:43.892389 ip-10-0-133-27 kubenswrapper[4917]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:43.905621 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905528    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.905621 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905616    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.905621 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905619    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.905621 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905623    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.905621 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905626    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.905621 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905629    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905631    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905634    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905637    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905640    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905642    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905645    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905649    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905652    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905654    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905657    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905659    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905662    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905664    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905667    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905670    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905672    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905674    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905678    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905681    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.905856 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905683    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905686    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905689    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905691    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905694    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905697    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905699    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905702    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905705    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905707    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905710    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905712    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905715    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905717    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905719    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905722    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905725    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905729    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905734    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.906334 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905737    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905739    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905742    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905745    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905747    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905750    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905752    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905755    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905757    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905760    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905762    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905764    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905780    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905783    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905786    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905789    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905792    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905803    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905806    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905809    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.906799 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905812    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905815    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905818    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905821    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905823    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905826    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905828    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905831    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905833    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905836    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905838    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905841    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905843    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905847    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905849    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905852    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905854    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905857    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905859    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905862    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.907271 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905865    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905869    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905873    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905876    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905878    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905881    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905884    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905887    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905889    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905892    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905895    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905898    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905901    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905903    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905906    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905909    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905911    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905914    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905917    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.907741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905919    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905922    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.905925    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906675    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906681    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906685    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906687    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906690    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906693    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906696    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906699    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906702    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906705    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906707    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906710    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906713    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906715    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906718    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906720    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.908196 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906724    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906726    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906729    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906732    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906737    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906739    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906742    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906745    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906748    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906756    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906760    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906764    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906781    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906786    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906790    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906794    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906798    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906802    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906806    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906809    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.908682 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906811    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906814    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906817    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906820    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906822    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906825    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906827    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906830    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906832    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906835    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906838    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906840    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906844    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906846    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906849    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906851    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906854    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906857    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906859    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.909204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906862    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906864    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906866    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906870    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906872    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906891    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906895    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906898    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906901    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906904    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906906    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906909    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906912    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906914    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906917    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906919    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906922    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906925    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906927    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906930    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.909680 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906932    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906935    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906938    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906940    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906943    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906945    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906948    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906950    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906953    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906955    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906958    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906960    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906963    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906965    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906967    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906970    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906973    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906976    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906979    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906981    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.910224 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906984    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906986    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906990    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906994    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.906998    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907001    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907005    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907008    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907012    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907014    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907017    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907499    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907505    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907508    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907511    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907514    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907516    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907519    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907522    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.910693 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907524    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907527    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907529    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907532    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907534    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907537    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907539    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907542    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907544    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907547    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907551    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907553    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907556    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907558    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907561    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907563    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907566    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907568    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907570    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.911161 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907573    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907575    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907578    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907580    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907583    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907585    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907587    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907590    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907594    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907597    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907600    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907602    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907605    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907607    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907610    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907612    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907615    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907617    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907620    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907622    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.911618 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907624    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907627    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907629    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907632    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907635    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907638    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907640    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907642    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907645    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907647    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907650    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907652    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907655    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907657    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907659    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907662    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907664    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907667    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907670    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.912288 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907673    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907679    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907683    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907687    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907690    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907693    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907696    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907698    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907701    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907704    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907706    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907708    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907711    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907713    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907716    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907718    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907721    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907724    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907726    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.913110 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907729    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907731    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907734    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907737    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907739    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907742    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907745    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907747    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907750    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907752    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907755    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907757    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907762    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907764    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907781    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907784    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907787    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907790    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907792    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.913914 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907795    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.907797    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907898    4917 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907909    4917 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907919    4917 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907924    4917 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907929    4917 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907932    4917 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907939    4917 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907943    4917 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907946    4917 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907950    4917 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907954    4917 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907957    4917 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907960    4917 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907963    4917 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907966    4917 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907969    4917 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907972    4917 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907975    4917 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907978    4917 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907981    4917 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907984    4917 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907987    4917 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:41:43.914708 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907991    4917 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907995    4917 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.907997    4917 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908002    4917 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908005    4917 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908009    4917 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908012    4917 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908015    4917 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908018    4917 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908021    4917 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908026    4917 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908029    4917 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908032    4917 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908035    4917 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908038    4917 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908041    4917 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908045    4917 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908048    4917 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908051    4917 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908054    4917 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908057    4917 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908062    4917 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908065    4917 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908068    4917 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908071    4917 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:41:43.915734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908074    4917 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908077    4917 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908080    4917 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908083    4917 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908086    4917 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908089    4917 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908092    4917 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908096    4917 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908099    4917 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908102    4917 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908107    4917 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908110    4917 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908114    4917 flags.go:64] FLAG: --help="false"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908117    4917 flags.go:64] FLAG: --hostname-override="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908120    4917 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908123    4917 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908126    4917 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908129    4917 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908133    4917 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908135    4917 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908138    4917 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908141    4917 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908144    4917 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:41:43.916936 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908147    4917 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908150    4917 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908152    4917 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908155    4917 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908158    4917 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908161    4917 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908164    4917 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908168    4917 flags.go:64] FLAG: --lock-file=""
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908170    4917 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908173    4917 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908176    4917 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908182    4917 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908184    4917 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908188    4917 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908190    4917 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908193    4917 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908196    4917 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908200    4917 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908202    4917 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908209    4917 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908217    4917 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908226    4917 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908229    4917 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908232    4917 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908235    4917 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:41:43.917981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908238    4917 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908241    4917 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908244    4917 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908247    4917 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908273    4917 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908276    4917 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908279    4917 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908282    4917 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908285    4917 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908289    4917 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908292    4917 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908295    4917 flags.go:64] FLAG: --port="10250"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908298    4917 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908301    4917 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-03986a7e375fb731c"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908305    4917 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908308    4917 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908311    4917 flags.go:64] FLAG: --register-node="true"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908314    4917 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908318    4917 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908321    4917 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908324    4917 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908327    4917 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908331    4917 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908334    4917 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908337    4917 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908340    4917 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:41:43.919027 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908345    4917 flags.go:64] FLAG: --runonce="false"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908348    4917 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908354    4917 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908357    4917 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908361    4917 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908364    4917 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908367    4917 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908370    4917 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908373    4917 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908377    4917 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908380    4917 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908382    4917 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908385    4917 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908388    4917 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908392    4917 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908395    4917 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908400    4917 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908403    4917 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908406    4917 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908410    4917 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908413    4917 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908417    4917 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908420    4917 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908423    4917 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908426    4917 flags.go:64] FLAG: --v="2"
Oct 30 19:41:43.920124 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908430    4917 flags.go:64] FLAG: --version="false"
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908434    4917 flags.go:64] FLAG: --vmodule=""
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908438    4917 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908442    4917 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908595    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908599    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908602    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908607    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908610    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908614    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908616    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908619    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908622    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908624    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908627    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908629    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908633    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908635    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908638    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908640    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.921289 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908643    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908646    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908649    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908651    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908654    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908657    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908659    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908662    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908665    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908668    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908670    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908673    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908676    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908678    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908681    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908683    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908686    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908689    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908692    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908696    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.922056 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908698    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908703    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908705    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908708    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908711    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908713    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908716    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908719    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908721    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908724    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908726    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908729    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908732    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908735    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908737    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908740    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908743    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908745    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908748    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908750    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.922591 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908753    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908755    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908758    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908761    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908764    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908778    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908781    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908784    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908787    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908789    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908793    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908798    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908801    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908805    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908808    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908811    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908813    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908816    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908818    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.923146 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908821    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908824    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908826    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908829    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908831    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908835    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908839    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908842    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908845    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908848    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908851    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908853    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908857    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908859    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908862    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908865    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908868    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908870    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908873    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.923899 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908875    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908878    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908880    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908883    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908886    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908891    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908893    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908897    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908900    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908902    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908905    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908908    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.908935    4917 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908971    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908976    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908979    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.924787 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908982    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908985    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908987    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908990    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908992    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908995    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.908998    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909001    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909005    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909007    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909010    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909013    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909016    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909018    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909021    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909024    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909027    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909030    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909032    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.925296 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909035    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909037    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909042    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909045    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909047    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909052    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909056    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909060    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909064    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909068    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909071    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909075    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909079    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909083    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909087    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909091    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909096    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909099    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909101    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909104    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.925883 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909107    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909109    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909112    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909114    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909117    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909120    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909123    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909125    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909129    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909132    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909134    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909137    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909140    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909142    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909146    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909149    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909152    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909156    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909159    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909161    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.926386 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909164    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909167    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909170    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909173    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909176    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909178    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909180    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909183    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909186    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909189    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909191    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909193    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909196    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909198    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909201    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909204    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909207    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909209    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909212    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909215    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.927019 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909218    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909221    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909224    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909226    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909229    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909231    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909235    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909237    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909240    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909244    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909247    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909249    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909252    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909255    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909257    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909260    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909262    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909265    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909267    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.927483 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909270    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909274    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909278    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909283    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.909285    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.915030    4917 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.915052    4917 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.915078    4917 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.915086    4917 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915172    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915181    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915187    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915191    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915196    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915200    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915205    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915209    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915214    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915218    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.927992 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915222    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915227    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915231    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915235    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915240    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915244    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915249    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915253    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915257    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915262    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915267    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915271    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915275    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915280    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915284    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915288    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915292    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915296    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915300    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.928487 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915305    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915309    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915313    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915317    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915321    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915325    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915332    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915337    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915341    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915346    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915350    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915354    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915358    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915363    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915367    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915371    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915376    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915380    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915385    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.929067 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915389    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915393    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915397    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915401    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915405    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915410    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915414    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915419    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915423    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915428    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915432    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915436    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915440    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915444    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915449    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915453    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915457    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915462    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915466    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915470    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.929828 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915474    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915479    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915483    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915487    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915492    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915496    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915500    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915507    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915512    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915517    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915521    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915527    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915531    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915536    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915541    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915545    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915549    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915554    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915558    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.930349 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915562    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915567    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915570    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915574    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915579    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915583    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915587    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915591    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915595    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915600    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915604    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915608    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915612    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915617    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915621    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915626    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915630    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915634    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.930823 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915638    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.915676    4917 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915919    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915928    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915932    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915938    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915943    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915955    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915960    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915964    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915968    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915972    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915977    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915981    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915985    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915990    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.931236 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915994    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.915998    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916002    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916006    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916010    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916016    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916024    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916030    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916034    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916038    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916042    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916046    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916051    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916055    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916059    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916063    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916070    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916076    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.931616 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916080    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916084    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916088    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916092    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916098    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916102    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916109    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916113    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916117    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916121    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916126    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916130    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916134    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916138    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916143    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916147    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916151    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916156    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916161    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.932038 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916166    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916170    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916174    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916178    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916183    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916187    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916191    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916195    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916200    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916204    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916208    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916221    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916226    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916230    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916234    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916238    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916243    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916247    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916268    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.932536 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916272    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916278    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916282    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916286    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916291    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916295    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916299    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916303    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916307    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916311    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916315    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916320    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916324    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916328    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916332    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916337    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916341    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916345    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916350    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916354    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.933106 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916358    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916362    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916366    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916370    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916375    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916379    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916383    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916387    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916391    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916396    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916400    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916403    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916408    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916412    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916417    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916421    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.933814 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.916465    4917 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916520    4917 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916528    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916534    4917 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916538    4917 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916544    4917 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916549    4917 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916553    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916558    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916562    4917 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916569    4917 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916575    4917 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916580    4917 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916585    4917 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916589    4917 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916593    4917 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916598    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916602    4917 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916607    4917 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:43.934329 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916611    4917 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916616    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916620    4917 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916625    4917 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916629    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916633    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916638    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916642    4917 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916646    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916650    4917 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916656    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916662    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916667    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916672    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916676    4917 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916681    4917 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916685    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916689    4917 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916693    4917 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:43.934935 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916697    4917 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916702    4917 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916706    4917 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916711    4917 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916715    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916719    4917 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916723    4917 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916728    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916732    4917 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916736    4917 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916740    4917 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916747    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916751    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916756    4917 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916760    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916764    4917 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916783    4917 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916787    4917 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916791    4917 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916795    4917 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:43.935562 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916800    4917 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916804    4917 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916808    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916812    4917 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916816    4917 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916820    4917 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916825    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916829    4917 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916833    4917 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916838    4917 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916842    4917 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916846    4917 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916850    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916854    4917 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916859    4917 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916863    4917 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916867    4917 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916872    4917 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916876    4917 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916880    4917 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:43.936198 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916885    4917 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916889    4917 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916893    4917 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916900    4917 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916904    4917 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916908    4917 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916913    4917 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916917    4917 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916921    4917 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916926    4917 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916930    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916934    4917 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916938    4917 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916942    4917 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916946    4917 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916950    4917 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916954    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916958    4917 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916962    4917 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916967    4917 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:43.936718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916972    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916977    4917 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916982    4917 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916986    4917 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916990    4917 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916994    4917 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.916998    4917 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.917006    4917 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:43.917011    4917 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.917099    4917 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.918636    4917 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.918709    4917 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.918988    4917 server.go:1014] "Starting client certificate rotation"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.919125    4917 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.919169    4917 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:36:02 +0000 UTC" deadline="2028-05-01 16:31:55.324776057 +0000 UTC"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.919195    4917 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13172h50m11.405582432s"
Oct 30 19:41:43.937373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.919677    4917 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:41:43.937837 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.921445    4917 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:41:43.937837 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.923070    4917 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:41:43.937837 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.925229    4917 log.go:25] "Validated CRI v1 image API"
Oct 30 19:41:43.937837 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.925980    4917 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:41:43.937837 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.927925    4917 fs.go:135] Filesystem UUIDs: map[1d5b3fd0-82d8-43f5-aea4-b9746774141b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 99d03b13-6faa-4eb3-aa8f-e0e517b6730b:/dev/nvme0n1p3]
Oct 30 19:41:43.937996 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.927940    4917 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06/userdata/shm major:0 minor:56 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c7b219c376a83723daa231618c253ed4344742dc57ce49c6005efd0aeae4ca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c7b219c376a83723daa231618c253ed4344742dc57ce49c6005efd0aeae4ca7/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b8465855c918b693eb982a133e7a9a15f91fe05a0490dbccd060c16e3ca1090/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b8465855c918b693eb982a133e7a9a15f91fe05a0490dbccd060c16e3ca1090/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432/userdata/shm major:0 minor:66 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~projected/kube-api-access-lp6ff:{mountpoint:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~projected/kube-api-access-lp6ff major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aefa597-5836-435a-9bc8-b54a2908db1c/volumes/kubernetes.io~projected/kube-api-access-ms5cj:{mountpoint:/var/lib/kubelet/pods/0aefa597-5836-435a-9bc8-b54a2908db1c/volumes/kubernetes.io~projected/kube-api-access-ms5cj major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f76c795-db38-4b17-b2d5-c3c3d9905edc/volumes/kubernetes.io~projected/kube-api-access-fwxjr:{mountpoint:/var/lib/kubelet/pods/0f76c795-db38-4b17-b2d5-c3c3d9905edc/volumes/kubernetes.io~projected/kube-api-access-fwxjr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19670c03-8928-4c59-b153-b51d3a77f63e/volumes/kubernetes.io~projected/kube-api-access-zcw28:{mountpoint:/var/lib/kubelet/pods/19670c03-8928-4c59-b153-b51d3a77f63e/volumes/kubernetes.io~projected/kube-api-access-zcw28 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb/volumes/kubernetes.io~projected/kube-api-access-ngw5v:{mountpoint:/var/lib/kubelet/pods/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb/volumes/kubernetes.io~projected/kube-api-access-ngw5v major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/292c4af8-ff18-45ab-96b2-bd05430793cf/volumes/kubernetes.io~projected/kube-api-access-f8kw7:{mountpoint:/var/lib/kubelet/pods/292c4af8-ff18-45ab-96b2-bd05430793cf/volumes/kubernetes.io~projected/kube-api-access-f8kw7 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cc8f2c8-4792-4fbf-b958-655692c689c7/volumes/kubernetes.io~projected/kube-api-access-b9h95:{mountpoint:/var/lib/kubelet/pods/2cc8f2c8-4792-4fbf-b958-655692c689c7/volumes/kubernetes.io~projected/kube-api-access-b9h95 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ae3d8c-faa0-468e-a557-d9f870fad555/volumes/kubernetes.io~projected/kube-api-access-25rfv:{mountpoint:/var/lib/kubelet/pods/31ae3d8c-faa0-468e-a557-d9f870fad555/volumes/kubernetes.io~projected/kube-api-access-25rfv major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ae3d8c-faa0-468e-a557-d9f870fad555/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/31ae3d8c-faa0-468e-a557-d9f870fad555/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3/volumes/kubernetes.io~projected/kube-api-access-v468k:{mountpoint:/var/lib/kubelet/pods/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3/volumes/kubernetes.io~projected/kube-api-access-v468k major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49dc5027-2bf7-4365-a107-6dc6bf106382/volumes/kubernetes.io~projected/kube-api-access-rc4mt:{mountpoint:/var/lib/kubelet/pods/49dc5027-2bf7-4365-a107-6dc6bf106382/volumes/kubernetes.io~projected/kube-api-access-rc4mt major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volumes/kubernetes.io~projected/kube-api-access-grkc2:{mountpoint:/var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volumes/kubernetes.io~projected/kube-api-access-grkc2 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f1128e1-21b4-437a-a7f5-e94567bec795/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4f1128e1-21b4-437a-a7f5-e94567bec795/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fe3fb9e-3883-4afe-ad11-64cd33045396/volumes/kubernetes.io~projected/kube-api-access-lmj2j:{mountpoint:/var/lib/kubelet/pods/4fe3fb9e-3883-4afe-ad11-64cd33045396/volumes/kubernetes.io~projected/kube-api-access-lmj2j major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fe3fb9e-3883-4afe-ad11-64cd33045396/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4fe3fb9e-3883-4afe-ad11-64cd33045396/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56510de8-5bb1-45b5-a057-f83143d0d836/volumes/kubernetes.io~projected/kube-api-access-b69l5:{mountpoint:/var/lib/kubelet/pods/56510de8-5bb1-45b5-a057-f83143d0d836/volumes/kubernetes.io~projected/kube-api-access-b69l5 major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~projected/kube-api-access-2lnwh:{mountpoint:/var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~projected/kube-api-access-2lnwh major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~secret/default-certificate major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~secret/stats-auth major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5de6c843-5c4d-4de8-99ff-950dd189b6bd/volumes/kubernetes.io~projected/kube-api-access-866th:{mountpoint:/var/lib/kubelet/pods/5de6c843-5c4d-4de8-99ff-950dd189b6bd/volumes/kubernetes.io~projected/kube-api-access-866th major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/710b36be-68a9-4ed5-84c6-30d5e3dc6801/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/710b36be-68a9-4ed5-84c6-30d5e3dc6801/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae66fed-db02-4f76-99ec-a80127e2a76a/volumes/kubernetes.io~projected/kube-api-access-wmdjv:{mountpoint:/var/lib/kubelet/pods/7ae66fed-db02-4f76-99ec-a80127e2a76a/volumes/kubernetes.io~projected/kube-api-access-wmdjv major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83293a26-814d-4b38-bf8e-10b28947c6c2/volumes/kubernetes.io~projected/kube-api-access-9szpb:{mountpoint:/var/lib/kubelet/pods/83293a26-814d-4b38-bf8e-10b28947c6c2/volumes/kubernetes.io~projected/kube-api-access-9szpb major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83293a26-814d-4b38-bf8e-10b28947c6c2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/83293a26-814d-4b38-bf8e-10b28947c6c2/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87025e83-063c-4c2b-9ba2-e551bd098d29/volumes/kubernetes.io~projected/kube-api-access-nz5m4:{mountpoint:/var/lib/kubelet/pods/87025e83-063c-4c2b-9ba2-e551bd098d29/volumes/kubernetes.io~projected/kube-api-access-nz5m4 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e23665b-8521-44d0-b403-ff1fefae7d74/volumes/kubernetes.io~projected/kube-api-access-j5td9:{mountpoint:/var/lib/kubelet/pods/9e23665b-8521-44d0-b403-ff1fefae7d74/volumes/kubernetes.io~projected/kube-api-access-j5td9 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e23665b-8521-44d0-b403-ff1fefae7d74/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9e23665b-8521-44d0-b403-ff1fefae7d74/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0070d9d-efde-4889-87e9-39b47aefbe58/volumes/kubernetes.io~projected/kube-api-access-ft5s4:{mountpoint:/var/lib/kubelet/pods/a0070d9d-efde-4889-87e9-39b47aefbe58/volumes/kubernetes.io~projected/kube-api-access-ft5s4 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0070d9d-efde-4889-87e9-39b47aefbe58/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a0070d9d-efde-4889-87e9-39b47aefbe58/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d/volumes/kubernetes.io~projected/kube-api-access-x89kj:{mountpoint:/var/lib/kubelet/pods/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d/volumes/kubernetes.io~projected/kube-api-access-x89kj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~projected/kube-api-access-hrrlg:{mountpoint:/var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~projected/kube-api-access-hrrlg major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1efd70d-88c5-4c06-bea1-b998190bb385/volumes/kubernetes.io~projected/kube-api-access-fvtv7:{mountpoint:/var/lib/kubelet/pods/b1efd70d-88c5-4c06-bea1-b998190bb385/volumes/kubernetes.io~projected/kube-api-access-fvtv7 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f/volumes/kubernetes.io~projected/kube-api-access-nwqr7:{mountpoint:/var/lib/kubelet/pods/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f/volumes/kubernetes.io~projected/kube-api-access-nwqr7 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d94a8213-30da-4eef-9967-8afc5725b7db/volumes/kubernetes.io~projected/kube-api-access-7695w:{mountpoint:/var/lib/kubelet/pods/d94a8213-30da-4eef-9967-8afc5725b7db/volumes/kubernetes.io~projected/kube-api-access-7695w major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb6f0ca7-6e21-4447-a036-2c91f24e885f/volumes/kubernetes.io~projected/kube-api-access-9m9ct:{mountpoint:/var/lib/kubelet/pods/eb6f0ca7-6e21-4447-a036-2c91f24e885f/volumes/kubernetes.io~projected/kube-api-access-9m9ct major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef755a60-5e9e-4ba7-9305-e3efbc283bf6/volumes/kubernetes.io~projected/kube-api-access-fxrgh:{mountpoint:/var/lib/kubelet/pods/ef755a60-5e9e-4ba7-9305-e3efbc283bf6/volumes/kubernetes.io~projected/kube-api-access-fxrgh major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~projected/kube-api-access-54pgw:{mountpoint:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~projected/kube-api-access-54pgw major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0a51e9a8fec0bf02e3672ad10905a33015c4196bfee61f029f91cd56672a75a4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9d12de9eecd9560441cada9109b9ffcdf40585b2bf14df098a5d2b7337090931/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1d76193b18720697d1e925ce7623a3b4bf46bc4422651a5b3ae0ca455f35d727/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/156c427e1db0b7e8503bc8baea07621c542177c1c4b32e445a30ba1cce40508d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e87a62c55f7b68a7e3ffd7e9eff7859388d3d6e92928941e32b2c369152d6a39/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/2bdaf4d156234b8a84922d72a910aea73c7616aeb253400e222c8dcddb5403f0/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/624c23a586f41814f09c8c444346e16e0f815e0f4a061dac381470c2aba2734b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5ef2eb7840fa9b94bd03f2cfed2ee8ff1af8dddf0bca0994fb8596511b2c7fae/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/0f99aecc5fc70d5597d1eafacb5870309b81689b70effc4190d9c4bf1cc4dc09/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/f1c3a9ff8182521707925b28b66b5157b10647daf9d9387537444f50a805c3d5/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/1ff1a13f898f9d2e38dad2f1cd9b728b25fd8cffcb87f272d26a4d05257bf932/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/e2a8ce861336559acc3f4f2052d8f05ff3be0ba86aa92252bf1d3a86b5020bf1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/807d0bf685002496c9454c3b440058e48a92b1f7bc15c5311bd8a17af6dfc900/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/b29fb76dfea13e85aa3242c6ad16121c224bab63476d0d816560bfb28657be8c/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/876890805ea91089f8770af565d816d1b6d9e214f17ea99042ab74f439958502/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0f90e4f811066f367982f61f85fecba1c939f89f5ef3f4058d58e232e1728803/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/452ba1883c685bbc786aa29a293f6d1c0cde03af42d5d75c884becc64a621519/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/59b5af2b7ab6967903d27ff3f5f57410e9e6c393ecbd95122e5a000f3ae1379e/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/6b9dd6780ae060c186517ce89008ac479a275728da7e3f63703370c1296ffd21/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/540b6cdea6537563649c5ead8631ab41666607f1345acb8599b6823fcb22c366/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e03f4bf3d38b8671a7d2fa1c8ff7c3352421bf0ec70b4457653b787b1f02a8a9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/7f30e8f051fddeb673b830bd7ea03f237234400c231cc083d808f575c82f851c/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b4c86162c38c3287dfa3a2554b9546ff308d7da6318384737e42e20900f9945e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b2af368f2ff321ebcaa73d8370a9a07801d3e454a20ac8adca4551319dc6c85b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/408a61d4730b75427eccdd380944f60e34a3b86dffa60758b9bc88b469795398/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fc1b1787d2211753759c3e19101511c5df50662a9f3e7b50659ab374923f75ec/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/ebdd212b3beec4a74b3f41bc0a5bc098f25ee2fcabf71770ec2d8cbb97518427/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e9e62aec8176ae2e2f38a7f09270a7caef48f7784d5451ce2cf7d18acda30165/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/82dd61448eb8cddc2152706dc32ef98de79d9e962798cf0ae415388c4f6924ab/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/4abe32c613166b13d9d207d3cd75e78c302d0c2e54d3727edd02e44a67f91b5d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/0147106d98e04420dd7dcd5f11a87b75a40bacce35c205f8d67c63977cea72e9/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/6668451436a83dc0b6d7ca44f5d868b5ff02bd62b79ace6cc531c3895b114ba9/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/bb9f4d9d670d32b6ee0ce2c9b4a6d4fbdee9f007b0b4ceab07151e91327a02ef/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/3060ea57cbdb917af41ee2fe62048c1084899ee99dfb861b328cdaff62c34b98/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/938760553c0a0ea531474e40b6b015adf10583fc3e8262aafc210e38b0c9e164/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/785dd8075177feeca6bd0d20fab7d2c2a3180e7fe6478238f0e1450e3931a414/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/4da78f5c56c676b38b7f46bfe1fee6bc98b1b0c683a8d514992b809832d297ec/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/105a6827499cdae2fc5e834866821a6980de5a0f1a68ec089e6b9f1976414be1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/1529d3bcdafe3f2cba6e58d62fc5109da9094f715cd0a54a255514042e4d6f45/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/25e54ef52c09918fe496ae0b6fe10f5f521e65b892419ad29acb3bc802e8fbd5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/be2fff0868e5fc6d71d516018ff7087d2793726d16691bbbfea8f127d2e2d391/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/548b1bd9785e87e3c038bc8d99c5c3a3d6b30757fa34c5e44d7a575bd658f55d/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/92ac2aac6a893f5c24634def5096148b6d999f0115a1987776e5e95e950f06f0/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/cabbdd972a299d0c74df404556fcb5a3267a30ded03423ad6124f7891ce3099c/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/d630c34b32030f6bfe488b56feec14812e8f6b433e7ee1db02b6adf24061d031/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/14be11ef48306aff38f752a39ee232bdcda74ac342430bf2089213b65bc0b622/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/4e15b5f30eea3743f7b1ecf069778cb3544b38aca4ac4054568d32a0487cb49b/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/9afc872824f628b31b36eff3c2bc9d8cf2c50c18b5fed6cadab4ddeecf45518b/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/f8de624192c872cd456afca71792fd62b5d4926a9f4b32106e4241595c1f73ab/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/d9bca2725d46e07ec199e82db2fd681bffb3d65e95b589fcaf24f79028ffbaf2/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ac14d2c43c666ac1c171f0d02400d01bc17f89df8ea14692489ea076c1b4b0cb/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ec769f2e629783c3eefba5d03e18cae5a3ca6f5b5e996124c73fa681e4e87270/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/36b57b3bfab6c331bd91eecb4ac92bbf4225cea1d9bb0717c37efd4685ac38d4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/95a18980c5285c969b1b94d17a869f621c61edc1945c6b17f2225b083f9363e3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/67eaf868c38e5a8eb74b56d210c09e6fa6c44fde0129aa5b6c3792a3d1738d84/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.936759    4917 manager.go:217] Machine: {Timestamp:2026-10-30 19:41:43.936293744 +0000 UTC m=+0.100695377 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198963 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274b4129928ac65ddd82fd88fdc727 SystemUUID:ec274b41-2992-8ac6-5ddd-82fd88fdc727 BootID:64b9989a-a8cf-4083-b8ed-02a9545e5b9e Filesystems:[{Device:/run/containers/storage/overlay-containers/f57ffac26ec7bf866d92457bc20ac854e17faa30c921aba1b3dc916acebbe81e/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/434f0b4a37050daea77a1fbd7f9b9b75c0bacdcaf83b9c7959bc4a36f558bf2c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b8465855c918b693eb982a133e7a9a15f91fe05a0490dbccd060c16e3ca1090/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ae3d8c-faa0-468e-a557-d9f870fad555/volumes/kubernetes.io~projected/kube-api-access-25rfv DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef755a60-5e9e-4ba7-9305-e3efbc283bf6/volumes/kubernetes.io~projected/kube-api-access-fxrgh DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b5d55cd2d89b29bee5df52c686ef2f7fcd37e68a83e1094c9e0deaf4a941b06/userdata/shm DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bf7313daece14df07dcee035dd5c19bd447b9f9653da0fa9d410c52b9957f92/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83293a26-814d-4b38-bf8e-10b28947c6c2/volumes/kubernetes.io~projected/kube-api-access-9szpb DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb/volumes/kubernetes.io~projected/kube-api-access-ngw5v DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0070d9d-efde-4889-87e9-39b47aefbe58/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/19670c03-8928-4c59-b153-b51d3a77f63e/volumes/kubernetes.io~projected/kube-api-access-zcw28 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fe3fb9e-3883-4afe-ad11-64cd33045396/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e0f495765f6399dbcb83ae6445ccc4fc2e872e98d294be146b464897ef1bc4a/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad1c0be5990db6f61174a5b07fea8e51f3a68ed80be2de96212987937d566ae1/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae66fed-db02-4f76-99ec-a80127e2a76a/volumes/kubernetes.io~projected/kube-api-access-wmdjv DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e592794e5f4db3442324d82b3a9df578784eba6d83f0c59a499647589fe4e6fe/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87025e83-063c-4c2b-9ba2-e551bd098d29/volumes/kubernetes.io~projected/kube-api-access-nz5m4 DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a07fce9f503c0df30c841cc7efcfdc05f6bc6b01ea832f3d33edf9f534076f2/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c7b219c376a83723daa231618c253ed4344742dc57ce49c6005efd0aeae4ca7/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e501d8ca9571225d67b8a654422b5af9c8d74e89e788572d9abd79259036a42/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/beaac7246b05f226a25bd8fc005aa9e8fb49e0bf5d33de6396a43d917219313d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0aefa597-5836-435a-9bc8-b54a2908db1c/volumes/kubernetes.io~projected/kube-api-access-ms5cj DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae767808-d262-4e49-8754-7149f5ec3fd1/volumes/kubernetes.io~projected/kube-api-access-hrrlg DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d/volumes/kubernetes.io~projected/kube-api-access-x89kj DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~projected/kube-api-access-lp6ff DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/710b36be-68a9-4ed5-84c6-30d5e3dc6801/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83293a26-814d-4b38-bf8e-10b28947c6c2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a11370fada4d37ac31bfff5d99c7cf2937737598868860dd493a2cdb8cc6ebd/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/852e9df0996751d5640da43a80d1cd199553ad5a6be565e0d566bea79338d75c/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0e1eab0d06de7660b11c1b7922d1e2da0d8af8fed9c3efb61b7df0f2c5f6cf5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e23665b-8521-44d0-b403-ff1fefae7d74/volumes/kubernetes.io~projected/kube-api-access-j5td9 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3/volumes/kubernetes.io~projected/kube-api-access-v468k DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/807cb3da42ed02ea6f4b23591563c7abc79bd42dbddb5438ae5d55316208baad/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c98d6808e98cdba7c2a793e6db0664578dfae9d9452ab6ff565820f82272432/userdata/shm DeviceMajor:0 DeviceMinor:66 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~projected/kube-api-access-54pgw DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0070d9d-efde-4889-87e9-39b47aefbe58/volumes/kubernetes.io~projected/kube-api-access-ft5s4 DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49dc5027-2bf7-4365-a107-6dc6bf106382/volumes/kubernetes.io~projected/kube-api-access-rc4mt DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f76c795-db38-4b17-b2d5-c3c3d9905edc/volumes/kubernetes.io~projected/kube-api-access-fwxjr DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31ae3d8c-faa0-468e-a557-d9f870fad555/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f52f29c687759970966f42215cd66f4e20954bc04a5b05a04a7bc1b205501d6/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d94a8213-30da-4eef-9967-8afc5725b7db/volumes/kubernetes.io~projected/kube-api-access-7695w DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cc8f2c8-4792-4fbf-b958-655692c689c7/volumes/kubernetes.io~projected/kube-api-access-b9h95 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb6f0ca7-6e21-4447-a036-2c91f24e885f/volumes/kubernetes.io~projected/kube-api-access-9m9ct DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~projected/kube-api-access-2lnwh DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ab44ad56e48e919300b30a1b4ffa9731b586860250b62fa22310e07707320cb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f/volumes/kubernetes.io~projected/kube-api-access-nwqr7 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fe3fb9e-3883-4afe-ad11-64cd33045396/volumes/kubernetes.io~projected/kube-api-access-lmj2j DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e23665b-8521-44d0-b403-ff1fefae7d74/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4d555bfc-7445-4727-a110-ce66dae3ee49/volumes/kubernetes.io~projected/kube-api-access-grkc2 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f1128e1-21b4-437a-a7f5-e94567bec795/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f89ccdfd691c25515be910316ff6483d32263413ec1a708007d0b4b546e387b4/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d3792592ceac52ad6f1855acd3fadc5585170fd6226f873aa1207688adf2ab2/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1efd70d-88c5-4c06-bea1-b998190bb385/volumes/kubernetes.io~projected/kube-api-access-fvtv7 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5de6c843-5c4d-4de8-99ff-950dd189b6bd/volumes/kubernetes.io~projected/kube-api-access-866th DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56510de8-5bb1-45b5-a057-f83143d0d836/volumes/kubernetes.io~projected/kube-api-access-b69l5 DeviceMajor:0 DeviceMinor:68 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/292c4af8-ff18-45ab-96b2-bd05430793cf/volumes/kubernetes.io~projected/kube-api-access-f8kw7 DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b117693-4843-4840-a6f1-5567a04226e1/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0d3792592ceac52 MacAddress:7e:e3:f1:b7:75:30 Speed:10000 Mtu:8901} {Name:1e0f495765f6399 MacAddress:e6:5c:8b:ba:85:f3 Speed:10000 Mtu:8901} {Name:2f52f29c6877599 MacAddress:82:46:fd:0b:1e:06 Speed:10000 Mtu:8901} {Name:3a11370fada4d37 MacAddress:5e:5c:71:a3:64:67 Speed:10000 Mtu:8901} {Name:434f0b4a37050da MacAddress:a6:c8:24:32:88:9c Speed:10000 Mtu:8901} {Name:5c7b219c376a837 MacAddress:e6:8d:2f:12:d1:3b Speed:10000 Mtu:8901} {Name:6a07fce9f503c0d MacAddress:66:b1:06:c2:27:e4 Speed:10000 Mtu:8901} {Name:852e9df0996751d MacAddress:5e:11:a0:dc:b9:c8 Speed:10000 Mtu:8901} {Name:8b8465855c918b6 MacAddress:62:72:0a:3e:7c:38 Speed:10000 Mtu:8901} {Name:ad1c0be5990db6f MacAddress:ca:41:f8:74:6d:ea Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:31:33:03:4a:61 Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:79:ce:78:8a:bb Speed:0 Mtu:8901} {Name:d0e1eab0d06de76 MacAddress:96:a3:13:b6:40:d2 Speed:10000 Mtu:8901} {Name:e592794e5f4db34 MacAddress:d2:02:8d:0f:59:ba Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:31:33:03:4a:61 Speed:0 Mtu:9001} {Name:f57ffac26ec7bf8 MacAddress:aa:80:16:5e:1f:b9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a6:46:f4:75:13:0c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:9e:04:48:95:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937205    4917 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937256    4917 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937552    4917 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937577    4917 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-27.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937691    4917 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937699    4917 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937709    4917 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937728    4917 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937746    4917 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937898    4917 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.937998    4917 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938030    4917 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938039    4917 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938050    4917 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938058    4917 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:41:43.938197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938065    4917 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:41:43.938962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938704    4917 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:41:43.938962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.938715    4917 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:41:43.939204 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.939183    4917 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:41:43.951411 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951392    4917 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:41:43.951411 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951407    4917 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:41:43.951540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951530    4917 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:41:43.951736 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951721    4917 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:41:43.951784 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951754    4917 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:41:43.951898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951886    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:41:43.951932 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951908    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:41:43.951932 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951923    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951933    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951941    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951948    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951953    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951959    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951965    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951970    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951983    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:41:43.951997 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.951993    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:41:43.952276 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.952027    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:41:43.952276 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.952053    4917 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:41:43.955752 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.955738    4917 server.go:1263] "Started kubelet"
Oct 30 19:41:43.956443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.955834    4917 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:41:43.956399 ip-10-0-133-27 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:41:43.956953 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.956843    4917 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:41:43.956953 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.956939    4917 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:41:43.958203 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.958051    4917 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:41:43.961324 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.961308    4917 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:41:43.962806 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962787    4917 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:41:43.962806 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962798    4917 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962845    4917 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:03 +0000 UTC" deadline="2028-05-23 09:43:01.058624961 +0000 UTC"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962863    4917 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13694h1m17.095765162s"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962896    4917 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962903    4917 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:43.962918    4917 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.962971    4917 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.963733    4917 factory.go:55] Registering systemd factory
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.963752    4917 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.964132    4917 factory.go:153] Registering CRI-O factory
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.964146    4917 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.964220    4917 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.964248    4917 factory.go:103] Registering Raw factory
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.964284    4917 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:41:43.964761 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.964663    4917 manager.go:319] Starting recovery of all containers
Oct 30 19:41:43.965881 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965845    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965883    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56510de8-5bb1-45b5-a057-f83143d0d836" volumeName="kubernetes.io/projected/56510de8-5bb1-45b5-a057-f83143d0d836-kube-api-access-b69l5" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965901    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0070d9d-efde-4889-87e9-39b47aefbe58" volumeName="kubernetes.io/secret/a0070d9d-efde-4889-87e9-39b47aefbe58-serving-cert" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965913    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef755a60-5e9e-4ba7-9305-e3efbc283bf6" volumeName="kubernetes.io/configmap/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-config" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965925    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="292c4af8-ff18-45ab-96b2-bd05430793cf" volumeName="kubernetes.io/empty-dir/292c4af8-ff18-45ab-96b2-bd05430793cf-data-volume" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965936    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e23665b-8521-44d0-b403-ff1fefae7d74" volumeName="kubernetes.io/secret/9e23665b-8521-44d0-b403-ff1fefae7d74-serving-cert" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965948    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="292c4af8-ff18-45ab-96b2-bd05430793cf" volumeName="kubernetes.io/projected/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-api-access-f8kw7" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965959    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f1128e1-21b4-437a-a7f5-e94567bec795" volumeName="kubernetes.io/secret/4f1128e1-21b4-437a-a7f5-e94567bec795-agent-certs" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965975    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe3fb9e-3883-4afe-ad11-64cd33045396" volumeName="kubernetes.io/projected/4fe3fb9e-3883-4afe-ad11-64cd33045396-kube-api-access-lmj2j" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965986    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5de6c843-5c4d-4de8-99ff-950dd189b6bd" volumeName="kubernetes.io/empty-dir/5de6c843-5c4d-4de8-99ff-950dd189b6bd-tmp-dir" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.965998    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca" seLinuxMountContext=""
Oct 30 19:41:43.966007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966009    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966021    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cc8f2c8-4792-4fbf-b958-655692c689c7" volumeName="kubernetes.io/projected/2cc8f2c8-4792-4fbf-b958-655692c689c7-kube-api-access-b9h95" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966033    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3" volumeName="kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966047    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6f0ca7-6e21-4447-a036-2c91f24e885f" volumeName="kubernetes.io/projected/eb6f0ca7-6e21-4447-a036-2c91f24e885f-kube-api-access-9m9ct" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966057    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f76c795-db38-4b17-b2d5-c3c3d9905edc" volumeName="kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966069    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3" volumeName="kubernetes.io/projected/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-kube-api-access-v468k" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966085    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d555bfc-7445-4727-a110-ce66dae3ee49" volumeName="kubernetes.io/secret/4d555bfc-7445-4727-a110-ce66dae3ee49-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966098    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83293a26-814d-4b38-bf8e-10b28947c6c2" volumeName="kubernetes.io/projected/83293a26-814d-4b38-bf8e-10b28947c6c2-kube-api-access-9szpb" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966110    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0070d9d-efde-4889-87e9-39b47aefbe58" volumeName="kubernetes.io/projected/a0070d9d-efde-4889-87e9-39b47aefbe58-kube-api-access-ft5s4" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966120    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d" volumeName="kubernetes.io/projected/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kube-api-access-x89kj" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966130    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966147    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f1128e1-21b4-437a-a7f5-e94567bec795" volumeName="kubernetes.io/configmap/4f1128e1-21b4-437a-a7f5-e94567bec795-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966158    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b117693-4843-4840-a6f1-5567a04226e1" volumeName="kubernetes.io/projected/5b117693-4843-4840-a6f1-5567a04226e1-kube-api-access-2lnwh" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966170    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e23665b-8521-44d0-b403-ff1fefae7d74" volumeName="kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-config" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966182    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef755a60-5e9e-4ba7-9305-e3efbc283bf6" volumeName="kubernetes.io/projected/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-kube-api-access-fxrgh" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966212    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966224    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ae3d8c-faa0-468e-a557-d9f870fad555" volumeName="kubernetes.io/secret/31ae3d8c-faa0-468e-a557-d9f870fad555-serving-cert" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966235    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83293a26-814d-4b38-bf8e-10b28947c6c2" volumeName="kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966242    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0070d9d-efde-4889-87e9-39b47aefbe58" volumeName="kubernetes.io/configmap/a0070d9d-efde-4889-87e9-39b47aefbe58-config" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966249    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966256    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d555bfc-7445-4727-a110-ce66dae3ee49" volumeName="kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-env-overrides" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966263    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ae3d8c-faa0-468e-a557-d9f870fad555" volumeName="kubernetes.io/configmap/31ae3d8c-faa0-468e-a557-d9f870fad555-config" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966285    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d555bfc-7445-4727-a110-ce66dae3ee49" volumeName="kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966292    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83293a26-814d-4b38-bf8e-10b28947c6c2" volumeName="kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-snapshots" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966300    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966310    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966317    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ae3d8c-faa0-468e-a557-d9f870fad555" volumeName="kubernetes.io/projected/31ae3d8c-faa0-468e-a557-d9f870fad555-kube-api-access-25rfv" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966324    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b117693-4843-4840-a6f1-5567a04226e1" volumeName="kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-default-certificate" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966333    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae66fed-db02-4f76-99ec-a80127e2a76a" volumeName="kubernetes.io/projected/7ae66fed-db02-4f76-99ec-a80127e2a76a-kube-api-access-wmdjv" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966342    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966353    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe3fb9e-3883-4afe-ad11-64cd33045396" volumeName="kubernetes.io/secret/4fe3fb9e-3883-4afe-ad11-64cd33045396-cert" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966364    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56510de8-5bb1-45b5-a057-f83143d0d836" volumeName="kubernetes.io/empty-dir/56510de8-5bb1-45b5-a057-f83143d0d836-tmp-dir" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966371    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b117693-4843-4840-a6f1-5567a04226e1" volumeName="kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-stats-auth" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966378    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83293a26-814d-4b38-bf8e-10b28947c6c2" volumeName="kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-tmp" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966385    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae767808-d262-4e49-8754-7149f5ec3fd1" volumeName="kubernetes.io/projected/ae767808-d262-4e49-8754-7149f5ec3fd1-kube-api-access-hrrlg" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966392    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc01cb5a-688e-4b24-ab7b-8157db6ccc1f" volumeName="kubernetes.io/configmap/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-telemetry-config" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966399    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aefa597-5836-435a-9bc8-b54a2908db1c" volumeName="kubernetes.io/projected/0aefa597-5836-435a-9bc8-b54a2908db1c-kube-api-access-ms5cj" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966407    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f76c795-db38-4b17-b2d5-c3c3d9905edc" volumeName="kubernetes.io/projected/0f76c795-db38-4b17-b2d5-c3c3d9905edc-kube-api-access-fwxjr" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966414    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5de6c843-5c4d-4de8-99ff-950dd189b6bd" volumeName="kubernetes.io/configmap/5de6c843-5c4d-4de8-99ff-950dd189b6bd-config-volume" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966422    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83293a26-814d-4b38-bf8e-10b28947c6c2" volumeName="kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966432    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966444    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4b078f-2e4a-4c6c-a352-16e2a2912bcb" volumeName="kubernetes.io/projected/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-kube-api-access-ngw5v" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966453    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49dc5027-2bf7-4365-a107-6dc6bf106382" volumeName="kubernetes.io/configmap/49dc5027-2bf7-4365-a107-6dc6bf106382-serviceca" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966460    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966469    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0932c21f-7a18-45f9-b244-c915394bf4d6" volumeName="kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966498    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f76c795-db38-4b17-b2d5-c3c3d9905edc" volumeName="kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966509    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4b078f-2e4a-4c6c-a352-16e2a2912bcb" volumeName="kubernetes.io/configmap/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966521    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="710b36be-68a9-4ed5-84c6-30d5e3dc6801" volumeName="kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966538    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e807294-3926-40b1-9203-9cbaf6f2ea40" volumeName="kubernetes.io/configmap/8e807294-3926-40b1-9203-9cbaf6f2ea40-nginx-conf" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966545    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e23665b-8521-44d0-b403-ff1fefae7d74" volumeName="kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-trusted-ca" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966560    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e23665b-8521-44d0-b403-ff1fefae7d74" volumeName="kubernetes.io/projected/9e23665b-8521-44d0-b403-ff1fefae7d74-kube-api-access-j5td9" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966567    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1efd70d-88c5-4c06-bea1-b998190bb385" volumeName="kubernetes.io/projected/b1efd70d-88c5-4c06-bea1-b998190bb385-kube-api-access-fvtv7" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966574    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d555bfc-7445-4727-a110-ce66dae3ee49" volumeName="kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-config" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966586    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc01cb5a-688e-4b24-ab7b-8157db6ccc1f" volumeName="kubernetes.io/projected/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-kube-api-access-nwqr7" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966610    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d94a8213-30da-4eef-9967-8afc5725b7db" volumeName="kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966620    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966638    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff11cf2e-b741-4ebc-a137-631442b49d23" volumeName="kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966647    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f76c795-db38-4b17-b2d5-c3c3d9905edc" volumeName="kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966654    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19670c03-8928-4c59-b153-b51d3a77f63e" volumeName="kubernetes.io/projected/19670c03-8928-4c59-b153-b51d3a77f63e-kube-api-access-zcw28" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966665    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d555bfc-7445-4727-a110-ce66dae3ee49" volumeName="kubernetes.io/projected/4d555bfc-7445-4727-a110-ce66dae3ee49-kube-api-access-grkc2" seLinuxMountContext=""
Oct 30 19:41:43.966645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966672    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5de6c843-5c4d-4de8-99ff-950dd189b6bd" volumeName="kubernetes.io/projected/5de6c843-5c4d-4de8-99ff-950dd189b6bd-kube-api-access-866th" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966682    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87025e83-063c-4c2b-9ba2-e551bd098d29" volumeName="kubernetes.io/projected/87025e83-063c-4c2b-9ba2-e551bd098d29-kube-api-access-nz5m4" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966702    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae767808-d262-4e49-8754-7149f5ec3fd1" volumeName="kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-tuned" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966708    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49dc5027-2bf7-4365-a107-6dc6bf106382" volumeName="kubernetes.io/projected/49dc5027-2bf7-4365-a107-6dc6bf106382-kube-api-access-rc4mt" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966716    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3" volumeName="kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966724    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83293a26-814d-4b38-bf8e-10b28947c6c2" volumeName="kubernetes.io/secret/83293a26-814d-4b38-bf8e-10b28947c6c2-serving-cert" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966731    4917 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae767808-d262-4e49-8754-7149f5ec3fd1" volumeName="kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-tmp" seLinuxMountContext=""
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966739    4917 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:41:43.969125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.966744    4917 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:41:43.969609 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.969593    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/1.log"
Oct 30 19:41:43.973522 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.973448    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/1.log"
Oct 30 19:41:43.993140 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.993106    4917 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:41:43.994328 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.994303    4917 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:41:43.994414 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.994332    4917 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:41:43.994414 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:43.994355    4917 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:41:43.994414 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:43.994395    4917 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:41:44.006903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.006878    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/1.log"
Oct 30 19:41:44.007012 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.006916    4917 generic.go:356] "Generic (PLEG): container finished" podID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509" exitCode=255
Oct 30 19:41:44.018881 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.018791    4917 generic.go:356] "Generic (PLEG): container finished" podID="8723c555bc23b9ece9590b4168888e4f" containerID="f3d4ad7b52e3754b68d512dc0d15bf95f5756a1dd350a86d0b7fc23d7fbfa194" exitCode=0
Oct 30 19:41:44.042051 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.042020    4917 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="ee9e449e528cac299c509802af5fd7f331fc6cd8cb3382caea649cd34c89af75" exitCode=0
Oct 30 19:41:44.042051 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.042043    4917 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="859bd57a435ede42dc26946348a619c9a94e87086f9a2a8a3861eb115b6a2d25" exitCode=0
Oct 30 19:41:44.042051 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.042050    4917 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="b6b472d6dafa5c28d91e4772afcfeec7b1dc5559b17998a82b3145f9b5ce9eed" exitCode=0
Oct 30 19:41:44.042051 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.042055    4917 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="529e5c4b44b00fb2cce070e8b0472242bd4b3765fdb307f3b778cc53ab9883d3" exitCode=0
Oct 30 19:41:44.042264 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.042061    4917 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="2bc8196097c56876de66564670b8a8b16591c8062fa29617831bf4cea36d6165" exitCode=0
Oct 30 19:41:44.042264 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.042066    4917 generic.go:356] "Generic (PLEG): container finished" podID="0f76c795-db38-4b17-b2d5-c3c3d9905edc" containerID="252153c5cc5df9f762f900957795cfd6da543107574021f355bdfe53eb459bdd" exitCode=0
Oct 30 19:41:44.051687 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.051669    4917 manager.go:324] Recovery completed
Oct 30 19:41:44.063428 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.063405    4917 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found"
Oct 30 19:41:44.068353 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.068336    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.069473 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.069454    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.069582 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.069501    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.069582 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.069515    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.072009 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.071991    4917 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:41:44.072142 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072122    4917 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:41:44.072206 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072136    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.072206 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072156    4917 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:41:44.072389 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072371    4917 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:41:44.072443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072390    4917 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:41:44.072443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072415    4917 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:41:44.072443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072423    4917 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:41:44.072443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072431    4917 policy_none.go:50] "Start"
Oct 30 19:41:44.072613 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072453    4917 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:41:44.072613 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072466    4917 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:41:44.072711 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072686    4917 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:41:44.072711 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072697    4917 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:41:44.072711 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.072706    4917 policy_none.go:44] "Start"
Oct 30 19:41:44.073409 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.073379    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.073499 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.073414    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.073499 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.073432    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.077964 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.077939    4917 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:41:44.080531 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.080519    4917 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:41:44.080581 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.080544    4917 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:41:44.080581 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.080553    4917 server.go:87] "Starting device plugin registration server"
Oct 30 19:41:44.080881 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.080869    4917 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:41:44.080914 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.080884    4917 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:41:44.080978 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.080960    4917 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:41:44.081120 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.081083    4917 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:41:44.081204 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.081130    4917 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:41:44.081204 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.081138    4917 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:41:44.081303 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.081189    4917 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:41:44.081116307Z","UUID":"92c751f5-a6fa-4ccb-af29-06e4f1e3e43e","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:41:44.082639 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.082612    4917 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:41:44.082720 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.082663    4917 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-27.us-west-1.compute.internal\" not found"
Oct 30 19:41:44.082942 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.082926    4917 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:41:44.083023 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.082947    4917 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:41:44.094833 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.094800    4917 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"]
Oct 30 19:41:44.094914 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.094905    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.095630 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.095614    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.095713 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.095639    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.095713 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.095650    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.095939 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.095928    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.096081 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.096069    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.097144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097127    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.097144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097124    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.097262 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097155    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.097262 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097161    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.097262 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097166    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.097262 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097175    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.097385 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.097294    4917 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.097485 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097456    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"5d870f2d14f468b90253c9dd98ce2b16","Type":"ContainerStarted","Data":"670d1fafc73f14bbf092942c2b7cdceabd60d4a86988df8b9f6c5cdfe44d20ef"}
Oct 30 19:41:44.097518 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097497    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"5d870f2d14f468b90253c9dd98ce2b16","Type":"ContainerStarted","Data":"2f0597e312e315810c613597af005f58328823a4cfebaa8e05ea8dcec28db913"}
Oct 30 19:41:44.097518 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097516    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"8723c555bc23b9ece9590b4168888e4f","Type":"ContainerStarted","Data":"054e4178b8239862d642ab6beabb92993eafb365c7d825d2b985a10666124534"}
Oct 30 19:41:44.097592 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097528    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"8723c555bc23b9ece9590b4168888e4f","Type":"ContainerDied","Data":"f3d4ad7b52e3754b68d512dc0d15bf95f5756a1dd350a86d0b7fc23d7fbfa194"}
Oct 30 19:41:44.097592 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097534    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.097592 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.097553    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal" event={"ID":"8723c555bc23b9ece9590b4168888e4f","Type":"ContainerStarted","Data":"3f185d9a0af9f804d354ba174f344ccf660ebb6e708b0465faf1ea45639f7b7b"}
Oct 30 19:41:44.098213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.098197    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.098279 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.098222    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.098279 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.098237    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.098379 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.098367    4917 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-27.us-west-1.compute.internal\" not found" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.150632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.150603    4917 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:44.152126 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.152113    4917 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:44.164599 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.164583    4917 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.181736 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.181714    4917 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:44.182750 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.182733    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:44.182863 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.182781    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:44.182863 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.182799    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:44.182863 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.182824    4917 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.202561 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.202539    4917 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:44.203766 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.203751    4917 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:44.204193 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.204178    4917 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:41:44.232595 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.232568    4917 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:44.232664 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.232626    4917 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.232664 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.232642    4917 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.304564 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.304488    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.304564 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.304516    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.304564 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.304536    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d870f2d14f468b90253c9dd98ce2b16-config\") pod \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"5d870f2d14f468b90253c9dd98ce2b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.304925 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.304911    4917 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.304978 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.304970    4917 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.305524 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.305507    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 19:41:44.305568 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.305540    4917 setters.go:546] "Node became not ready" node="ip-10-0-133-27.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:41:44Z","lastTransitionTime":"2026-10-30T19:41:44Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 19:41:44.324008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.323988    4917 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-27.us-west-1.compute.internal" is owned by different node (oldNodeID="64939808-0156-4b4f-a4c4-89e37a9bb48f", newNodeID=""), cleaning up...
Oct 30 19:41:44.326590 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.326576    4917 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:44.326630 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:44.326615    4917 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.404962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.404931    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.404962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.404962    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d870f2d14f468b90253c9dd98ce2b16-config\") pod \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"5d870f2d14f468b90253c9dd98ce2b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.405144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.404979    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.405144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.405023    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.405144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.405041    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d870f2d14f468b90253c9dd98ce2b16-config\") pod \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"5d870f2d14f468b90253c9dd98ce2b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.405144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.405017    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8723c555bc23b9ece9590b4168888e4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" (UID: \"8723c555bc23b9ece9590b4168888e4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:44.938934 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:44.938910    4917 apiserver.go:52] "Watching apiserver"
Oct 30 19:41:45.046906 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.046875    4917 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:45.048084 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.048057    4917 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:45.095457 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.095427    4917 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:41:45.096126 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.096102    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-pt4mt","openshift-insights/insights-runtime-extractor-nwlvj","openshift-network-operator/iptables-alerter-d88b4","openshift-ingress-canary/ingress-canary-9n45f","openshift-network-console/networking-console-plugin-86c478bb88-csrgl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8","openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv","openshift-dns/node-resolver-57p4t","kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7","openshift-image-registry/image-registry-7b45499cc5-52dbl","openshift-image-registry/node-ca-t7wjp","kube-system/global-pull-secret-syncer-x6kbk","openshift-console-operator/console-operator-7cd56b8dbc-pjlwh","openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d","openshift-image-registry/image-registry-6d57c9fd74-7lvhr","openshift-ovn-kubernetes/ovnkube-node-4qcq7","openshift-multus/network-metrics-daemon-tfswf","openshift-dns/dns-default-9wckj","openshift-cluster-node-tuning-operator/tuned-mrdzz","openshift-multus/multus-cn92p","openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p","kube-system/kas-connection-checker-685895c7c9-txjbg","openshift-ingress/router-default-569cc6f5cf-pw4pl","kube-system/konnectivity-agent-tznqp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal","openshift-insights/insights-operator-7c4986d449-t9qb9","kube-system/kas-connection-checker-685895c7c9-8wclq","openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf","kube-system/kas-connection-checker-685895c7c9-swx4r","openshift-multus/multus-additional-cni-plugins-lngvb"]
Oct 30 19:41:45.097991 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.097966    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.098732 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.098716    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.099407 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.099391    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.099716 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.099696    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.100839 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.100823    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:45.102249 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.102232    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:45.107621 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.107604    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.109646 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.109626    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.110492 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.110468    4917 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:45.110585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.110553    4917 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:45.111008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.110988    4917 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:45.111069 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.111041    4917 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-27.us-west-1.compute.internal"
Oct 30 19:41:45.113921 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.113903    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.162005 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.161985    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:41:45.162123 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162008    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162123 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162072    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162217 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162206    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:41:45.162259 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162249    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.162417 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162394    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.162462 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162429    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162462 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162452    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162542 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162529    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.162691 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162677    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.162748 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162698    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162748 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162725    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162834 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162764    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.162834 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162811    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.162834 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162818    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:41:45.162917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162836    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:41:45.162917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162843    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:41:45.162917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162909    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:41:45.163021 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.162968    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.163133 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.163120    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.163238 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.163225    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.163285 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.163260    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:41:45.163998 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.163983    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.164075 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.164025    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.164075 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.164032    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.164283 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.164266    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.165369 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.165353    4917 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:41:45.210547 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210485    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-netns\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.210547 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210512    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-bin\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.210547 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210530    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/292c4af8-ff18-45ab-96b2-bd05430793cf-crio-socket\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.210547 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210546    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-device-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.210741 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210561    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-systemd\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.210741 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210607    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-run\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.210741 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210637    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rfv\" (UniqueName: \"kubernetes.io/projected/31ae3d8c-faa0-468e-a557-d9f870fad555-kube-api-access-25rfv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:45.210741 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210657    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-config\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.210741 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210688    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-env-overrides\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.210929 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210880    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-iptables-alerter-script\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.210929 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210909    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.211004 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210933    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkc2\" (UniqueName: \"kubernetes.io/projected/4d555bfc-7445-4727-a110-ce66dae3ee49-kube-api-access-grkc2\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.211004 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210949    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.211004 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.210995    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.211118 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211066    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cnibin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.211118 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211089    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdjv\" (UniqueName: \"kubernetes.io/projected/7ae66fed-db02-4f76-99ec-a80127e2a76a-kube-api-access-wmdjv\") pod \"kas-connection-checker-685895c7c9-txjbg\" (UID: \"7ae66fed-db02-4f76-99ec-a80127e2a76a\") " pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:45.211208 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211134    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw5v\" (UniqueName: \"kubernetes.io/projected/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-kube-api-access-ngw5v\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.211208 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211171    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.211208 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211198    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmj2j\" (UniqueName: \"kubernetes.io/projected/4fe3fb9e-3883-4afe-ad11-64cd33045396-kube-api-access-lmj2j\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:45.211339 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211222    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v468k\" (UniqueName: \"kubernetes.io/projected/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-kube-api-access-v468k\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.211339 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211248    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-etc-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.211339 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211276    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.211339 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211311    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-etc-kubernetes\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.211487 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211364    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e807294-3926-40b1-9203-9cbaf6f2ea40-nginx-conf\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:45.211487 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211395    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.211487 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211422    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.211487 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211453    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.211622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211494    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:45.211622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211524    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.211622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211526    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysconfig\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.211622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211573    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.211622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211590    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5cj\" (UniqueName: \"kubernetes.io/projected/0aefa597-5836-435a-9bc8-b54a2908db1c-kube-api-access-ms5cj\") pod \"volume-data-source-validator-58c9f5f46d-768l8\" (UID: \"0aefa597-5836-435a-9bc8-b54a2908db1c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:45.211622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211607    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e807294-3926-40b1-9203-9cbaf6f2ea40-nginx-conf\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:45.211885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211650    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-default-certificate\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.211885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211675    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-ovn\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.211885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211749    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5td9\" (UniqueName: \"kubernetes.io/projected/9e23665b-8521-44d0-b403-ff1fefae7d74-kube-api-access-j5td9\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.211885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211795    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-socket-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.211885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211822    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49dc5027-2bf7-4365-a107-6dc6bf106382-serviceca\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.211885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211849    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-k8s-cni-cncf-io\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.212101 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211882    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.212101 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211936    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-kubelet\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.212101 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.211972    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-systemd-units\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.212101 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212011    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.212101 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212038    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-service-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212135    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxjr\" (UniqueName: \"kubernetes.io/projected/0f76c795-db38-4b17-b2d5-c3c3d9905edc-kube-api-access-fwxjr\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212167    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212194    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212227    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-conf-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212275    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49dc5027-2bf7-4365-a107-6dc6bf106382-serviceca\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212293    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.212321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212309    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-service-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212289    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212319    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5s4\" (UniqueName: \"kubernetes.io/projected/a0070d9d-efde-4889-87e9-39b47aefbe58-kube-api-access-ft5s4\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212373    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-netd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212389    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-host\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212407    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4mt\" (UniqueName: \"kubernetes.io/projected/49dc5027-2bf7-4365-a107-6dc6bf106382-kube-api-access-rc4mt\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212428    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212464    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/292c4af8-ff18-45ab-96b2-bd05430793cf-data-volume\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212565    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/292c4af8-ff18-45ab-96b2-bd05430793cf-data-volume\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212558    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5m4\" (UniqueName: \"kubernetes.io/projected/87025e83-063c-4c2b-9ba2-e551bd098d29-kube-api-access-nz5m4\") pod \"network-check-source-6f78d59f99-69s2p\" (UID: \"87025e83-063c-4c2b-9ba2-e551bd098d29\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212609    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-host-slash\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.212632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212631    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212652    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212694    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212728    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212755    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-kubelet-config\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212800    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-slash\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212833    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212847    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-registration-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212897    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-conf\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212928    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-var-lib-kubelet\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.212936    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213006    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-dbus\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.213042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213033    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-log-socket\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213060    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213093    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-system-cni-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213137    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-bin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213199    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d555bfc-7445-4727-a110-ce66dae3ee49-ovn-node-metrics-cert\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213244    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-lib-modules\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213290    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-tuned\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213352    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49dc5027-2bf7-4365-a107-6dc6bf106382-host\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213382    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-tuned\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213401    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-netns\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213465    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213493    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-config\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.213523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213521    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-sys\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213544    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-hostroot\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213568    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-tmp\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213570    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213617    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-tmp\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213646    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-multus-certs\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213677    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-config\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213694    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-stats-auth\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213743    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866th\" (UniqueName: \"kubernetes.io/projected/5de6c843-5c4d-4de8-99ff-950dd189b6bd-kube-api-access-866th\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213791    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83293a26-814d-4b38-bf8e-10b28947c6c2-serving-cert\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213882    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213901    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-kubernetes\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213915    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213931    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szpb\" (UniqueName: \"kubernetes.io/projected/83293a26-814d-4b38-bf8e-10b28947c6c2-kube-api-access-9szpb\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213943    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:41:45.213986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.213971    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56510de8-5bb1-45b5-a057-f83143d0d836-tmp-dir\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214004    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214027    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214043    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcw28\" (UniqueName: \"kubernetes.io/projected/19670c03-8928-4c59-b153-b51d3a77f63e-kube-api-access-zcw28\") pod \"kas-connection-checker-685895c7c9-8wclq\" (UID: \"19670c03-8928-4c59-b153-b51d3a77f63e\") " pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214048    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56510de8-5bb1-45b5-a057-f83143d0d836-tmp-dir\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214061    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrgh\" (UniqueName: \"kubernetes.io/projected/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-kube-api-access-fxrgh\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214097    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56510de8-5bb1-45b5-a057-f83143d0d836-hosts-file\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214141    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31ae3d8c-faa0-468e-a557-d9f870fad555-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214188    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214189    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-systemd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214223    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214248    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5de6c843-5c4d-4de8-99ff-950dd189b6bd-config-volume\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214303    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214320    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cni-binary-copy\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214341    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214349    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-socket-dir-parent\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214377    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0070d9d-efde-4889-87e9-39b47aefbe58-config\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214465    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31ae3d8c-faa0-468e-a557-d9f870fad555-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:45.214534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214527    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214556    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214586    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214619    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89kj\" (UniqueName: \"kubernetes.io/projected/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kube-api-access-x89kj\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214639    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214647    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214643    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214648    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnwh\" (UniqueName: \"kubernetes.io/projected/5b117693-4843-4840-a6f1-5567a04226e1-kube-api-access-2lnwh\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214667    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0070d9d-efde-4889-87e9-39b47aefbe58-config\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214662    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31ae3d8c-faa0-468e-a557-d9f870fad555-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214704    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214709    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214728    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214730    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214748    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hjc6w\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214739    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214799    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214833    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214836    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214849    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214835    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214753    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214869    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214903    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214909    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-var-lib-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214944    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214950    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214964    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-trusted-ca\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.214998    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-sys-fs\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215061    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-tmp\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215143    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-kubelet\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215152    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae767808-d262-4e49-8754-7149f5ec3fd1-tmp\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215173    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-daemon-config\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215190    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215207    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69l5\" (UniqueName: \"kubernetes.io/projected/56510de8-5bb1-45b5-a057-f83143d0d836-kube-api-access-b69l5\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215222    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-system-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215239    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215280    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f1128e1-21b4-437a-a7f5-e94567bec795-konnectivity-ca\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215296    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e23665b-8521-44d0-b403-ff1fefae7d74-trusted-ca\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215311    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de6c843-5c4d-4de8-99ff-950dd189b6bd-tmp-dir\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215364    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e23665b-8521-44d0-b403-ff1fefae7d74-serving-cert\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.215358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215386    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de6c843-5c4d-4de8-99ff-950dd189b6bd-tmp-dir\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215397    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9ct\" (UniqueName: \"kubernetes.io/projected/eb6f0ca7-6e21-4447-a036-2c91f24e885f-kube-api-access-9m9ct\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215427    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-modprobe-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215455    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215486    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215521    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqr7\" (UniqueName: \"kubernetes.io/projected/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-kube-api-access-nwqr7\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215528    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e23665b-8521-44d0-b403-ff1fefae7d74-serving-cert\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215549    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215560    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83293a26-814d-4b38-bf8e-10b28947c6c2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215575    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtv7\" (UniqueName: \"kubernetes.io/projected/b1efd70d-88c5-4c06-bea1-b998190bb385-kube-api-access-fvtv7\") pod \"migrator-66df5d496b-kpp2d\" (UID: \"b1efd70d-88c5-4c06-bea1-b998190bb385\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215603    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215628    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-script-lib\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215656    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kw7\" (UniqueName: \"kubernetes.io/projected/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-api-access-f8kw7\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215656    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215680    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/ae767808-d262-4e49-8754-7149f5ec3fd1-kube-api-access-hrrlg\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215703    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-multus\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215726    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cnibin\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215743    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215736    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215760    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215818    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215878    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215921    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-node-log\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.215962    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-snapshots\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216019    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe3fb9e-3883-4afe-ad11-64cd33045396-cert\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216030    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/83293a26-814d-4b38-bf8e-10b28947c6c2-snapshots\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216040    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f1128e1-21b4-437a-a7f5-e94567bec795-agent-certs\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216060    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h95\" (UniqueName: \"kubernetes.io/projected/2cc8f2c8-4792-4fbf-b958-655692c689c7-kube-api-access-b9h95\") pod \"kas-connection-checker-685895c7c9-swx4r\" (UID: \"2cc8f2c8-4792-4fbf-b958-655692c689c7\") " pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216061    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216078    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0070d9d-efde-4889-87e9-39b47aefbe58-serving-cert\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:45.216782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.216222    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0070d9d-efde-4889-87e9-39b47aefbe58-serving-cert\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:45.221109 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.221093    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-config\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.221230 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.221217    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.221844 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.221830    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.222008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.221995    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-default-certificate\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.223898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.223881    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.223972 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.223959    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83293a26-814d-4b38-bf8e-10b28947c6c2-serving-cert\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.224023 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.224005    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-stats-auth\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.224056 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.224010    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.225035 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.225021    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31ae3d8c-faa0-468e-a557-d9f870fad555-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:45.225098 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.225083    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.258602 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.258574    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:41:45.258602 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.258580    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:41:45.259541 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259524    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:41:45.259612 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259599    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:41:45.259734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259718    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:41:45.259801 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259739    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:41:45.259801 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259759    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wfbg\""
Oct 30 19:41:45.259801 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259797    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgpv7\""
Oct 30 19:41:45.259923 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.259912    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\""
Oct 30 19:41:45.260115 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260100    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjk7j\""
Oct 30 19:41:45.260163 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260113    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pcf9z\""
Oct 30 19:41:45.260163 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260150    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:41:45.260231 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260198    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqb6z\""
Oct 30 19:41:45.260324 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260253    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\""
Oct 30 19:41:45.260357 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260297    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shc6v\""
Oct 30 19:41:45.260385 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260364    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v4lmq\""
Oct 30 19:41:45.260385 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260366    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pxncz\""
Oct 30 19:41:45.260916 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260896    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhxv\""
Oct 30 19:41:45.261004 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260923    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:41:45.261004 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.260987    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47f99\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261417    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261530    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261578    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261626    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261687    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261645    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bk6w\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261914    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261967    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsfbm\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261920    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.261639    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262115    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262278    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262297    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262346    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262391    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262405    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262427    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262436    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262466    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.262607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262621    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262719    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262804    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262913    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.262986    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263095    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6h9\""
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263152    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpdkk\""
Oct 30 19:41:45.263216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263185    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.263516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263323    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.263516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263339    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.263516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263458    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:41:45.263516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263489    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.263516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263499    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\""
Oct 30 19:41:45.263727 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263565    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:45.263727 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.263594    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d555bfc-7445-4727-a110-ce66dae3ee49-ovn-node-metrics-cert\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.265331 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265309    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.265331 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265326    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5de6c843-5c4d-4de8-99ff-950dd189b6bd-config-volume\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.265507 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265379    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cni-binary-copy\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.265558 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265525    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-daemon-config\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.265558 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265550    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f1128e1-21b4-437a-a7f5-e94567bec795-konnectivity-ca\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:45.265910 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265892    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710b36be-68a9-4ed5-84c6-30d5e3dc6801-original-pull-secret\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.265968 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.265926    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-ovnkube-script-lib\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.267223 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.267204    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f1128e1-21b4-437a-a7f5-e94567bec795-agent-certs\") pod \"konnectivity-agent-tznqp\" (UID: \"4f1128e1-21b4-437a-a7f5-e94567bec795\") " pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:45.267223 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.267217    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe3fb9e-3883-4afe-ad11-64cd33045396-cert\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:45.271114 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.271097    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d555bfc-7445-4727-a110-ce66dae3ee49-env-overrides\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.271202 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.271181    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-iptables-alerter-script\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.274315 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.274295    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rfv\" (UniqueName: \"kubernetes.io/projected/31ae3d8c-faa0-468e-a557-d9f870fad555-kube-api-access-25rfv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rh65c\" (UID: \"31ae3d8c-faa0-468e-a557-d9f870fad555\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c"
Oct 30 19:41:45.285469 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.285449    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdjv\" (UniqueName: \"kubernetes.io/projected/7ae66fed-db02-4f76-99ec-a80127e2a76a-kube-api-access-wmdjv\") pod \"kas-connection-checker-685895c7c9-txjbg\" (UID: \"7ae66fed-db02-4f76-99ec-a80127e2a76a\") " pod="kube-system/kas-connection-checker-685895c7c9-txjbg"
Oct 30 19:41:45.285717 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.285699    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.285827 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.285811    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkc2\" (UniqueName: \"kubernetes.io/projected/4d555bfc-7445-4727-a110-ce66dae3ee49-kube-api-access-grkc2\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.287664 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.287648    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxjr\" (UniqueName: \"kubernetes.io/projected/0f76c795-db38-4b17-b2d5-c3c3d9905edc-kube-api-access-fwxjr\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.294993 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.294978    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.305042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.305025    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.305229 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.305212    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqr7\" (UniqueName: \"kubernetes.io/projected/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-kube-api-access-nwqr7\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.305819 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.305802    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrgh\" (UniqueName: \"kubernetes.io/projected/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-kube-api-access-fxrgh\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.306137 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.306116    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw5v\" (UniqueName: \"kubernetes.io/projected/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-kube-api-access-ngw5v\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.306192 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.306177    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kw7\" (UniqueName: \"kubernetes.io/projected/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-api-access-f8kw7\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.306259 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.306243    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5m4\" (UniqueName: \"kubernetes.io/projected/87025e83-063c-4c2b-9ba2-e551bd098d29-kube-api-access-nz5m4\") pod \"network-check-source-6f78d59f99-69s2p\" (UID: \"87025e83-063c-4c2b-9ba2-e551bd098d29\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-69s2p"
Oct 30 19:41:45.307749 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.307723    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5s4\" (UniqueName: \"kubernetes.io/projected/a0070d9d-efde-4889-87e9-39b47aefbe58-kube-api-access-ft5s4\") pod \"service-ca-operator-544d749c8c-56vpv\" (UID: \"a0070d9d-efde-4889-87e9-39b47aefbe58\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv"
Oct 30 19:41:45.307833 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.307732    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtv7\" (UniqueName: \"kubernetes.io/projected/b1efd70d-88c5-4c06-bea1-b998190bb385-kube-api-access-fvtv7\") pod \"migrator-66df5d496b-kpp2d\" (UID: \"b1efd70d-88c5-4c06-bea1-b998190bb385\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d"
Oct 30 19:41:45.307833 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.307794    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnwh\" (UniqueName: \"kubernetes.io/projected/5b117693-4843-4840-a6f1-5567a04226e1-kube-api-access-2lnwh\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.307902 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.307888    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szpb\" (UniqueName: \"kubernetes.io/projected/83293a26-814d-4b38-bf8e-10b28947c6c2-kube-api-access-9szpb\") pod \"insights-operator-7c4986d449-t9qb9\" (UID: \"83293a26-814d-4b38-bf8e-10b28947c6c2\") " pod="openshift-insights/insights-operator-7c4986d449-t9qb9"
Oct 30 19:41:45.307932 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.307916    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/ae767808-d262-4e49-8754-7149f5ec3fd1-kube-api-access-hrrlg\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.313036 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.312966    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcw28\" (UniqueName: \"kubernetes.io/projected/19670c03-8928-4c59-b153-b51d3a77f63e-kube-api-access-zcw28\") pod \"kas-connection-checker-685895c7c9-8wclq\" (UID: \"19670c03-8928-4c59-b153-b51d3a77f63e\") " pod="kube-system/kas-connection-checker-685895c7c9-8wclq"
Oct 30 19:41:45.313036 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.312990    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89kj\" (UniqueName: \"kubernetes.io/projected/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kube-api-access-x89kj\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.313036 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.312997    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4mt\" (UniqueName: \"kubernetes.io/projected/49dc5027-2bf7-4365-a107-6dc6bf106382-kube-api-access-rc4mt\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.313213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313073    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h95\" (UniqueName: \"kubernetes.io/projected/2cc8f2c8-4792-4fbf-b958-655692c689c7-kube-api-access-b9h95\") pod \"kas-connection-checker-685895c7c9-swx4r\" (UID: \"2cc8f2c8-4792-4fbf-b958-655692c689c7\") " pod="kube-system/kas-connection-checker-685895c7c9-swx4r"
Oct 30 19:41:45.313213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313151    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v468k\" (UniqueName: \"kubernetes.io/projected/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-kube-api-access-v468k\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.313213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313159    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.313213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313177    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7695w\" (UniqueName: \"kubernetes.io/projected/d94a8213-30da-4eef-9967-8afc5725b7db-kube-api-access-7695w\") pod \"network-check-target-pt4mt\" (UID: \"d94a8213-30da-4eef-9967-8afc5725b7db\") " pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:45.313213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313206    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5td9\" (UniqueName: \"kubernetes.io/projected/9e23665b-8521-44d0-b403-ff1fefae7d74-kube-api-access-j5td9\") pod \"console-operator-7cd56b8dbc-pjlwh\" (UID: \"9e23665b-8521-44d0-b403-ff1fefae7d74\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:45.313400 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313385    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9ct\" (UniqueName: \"kubernetes.io/projected/eb6f0ca7-6e21-4447-a036-2c91f24e885f-kube-api-access-9m9ct\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:45.313991 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.313973    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69l5\" (UniqueName: \"kubernetes.io/projected/56510de8-5bb1-45b5-a057-f83143d0d836-kube-api-access-b69l5\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.315394 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.315378    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5cj\" (UniqueName: \"kubernetes.io/projected/0aefa597-5836-435a-9bc8-b54a2908db1c-kube-api-access-ms5cj\") pod \"volume-data-source-validator-58c9f5f46d-768l8\" (UID: \"0aefa597-5836-435a-9bc8-b54a2908db1c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-768l8"
Oct 30 19:41:45.316057 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316042    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmj2j\" (UniqueName: \"kubernetes.io/projected/4fe3fb9e-3883-4afe-ad11-64cd33045396-kube-api-access-lmj2j\") pod \"ingress-canary-9n45f\" (UID: \"4fe3fb9e-3883-4afe-ad11-64cd33045396\") " pod="openshift-ingress-canary/ingress-canary-9n45f"
Oct 30 19:41:45.316816 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316801    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56510de8-5bb1-45b5-a057-f83143d0d836-hosts-file\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.316859 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316823    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-systemd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.316859 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316845    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.316955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316859    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-socket-dir-parent\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.316955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316903    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-socket-dir-parent\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.316955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316914    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56510de8-5bb1-45b5-a057-f83143d0d836-hosts-file\") pod \"node-resolver-57p4t\" (UID: \"56510de8-5bb1-45b5-a057-f83143d0d836\") " pod="openshift-dns/node-resolver-57p4t"
Oct 30 19:41:45.316955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316917    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.316955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316951    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-systemd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316958    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.316991    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317020    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-var-lib-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317046    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-sys-fs\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317068    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-kubelet\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317079    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-var-lib-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317091    4917 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317093    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317124    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-system-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317126    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-kubelet\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317139    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-sys-fs\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317152    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-modprobe-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317091    4917 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317185    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817164387 +0000 UTC m=+1.981566025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317188    4917 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317206    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817192398 +0000 UTC m=+1.981594018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:45.317224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317222    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317244    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817229128 +0000 UTC m=+1.981630777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317187    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-system-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317266    4917 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317264    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-modprobe-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317278    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-multus\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317300    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.81729287 +0000 UTC m=+1.981694527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317303    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-multus\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317315    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cnibin\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317339    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317362    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317375    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-cnibin\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317385    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317408    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817399418 +0000 UTC m=+1.981801072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317429    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317455    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-node-log\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317468    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317477    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317480    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-netns\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317494    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-bin\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317504    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817496948 +0000 UTC m=+1.981898596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317520    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-bin\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317528    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/292c4af8-ff18-45ab-96b2-bd05430793cf-crio-socket\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317542    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-node-log\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317545    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-run-netns\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317552    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-device-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317581    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-systemd\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317588    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-device-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317582    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/292c4af8-ff18-45ab-96b2-bd05430793cf-crio-socket\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317602    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-run\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317623    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-systemd\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317631    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-run\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317638    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317583    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866th\" (UniqueName: \"kubernetes.io/projected/5de6c843-5c4d-4de8-99ff-950dd189b6bd-kube-api-access-866th\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317667    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317694    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cnibin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317698    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-cni-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317722    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-etc-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317744    4917 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317748    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317766    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-cnibin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317788    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-etc-kubernetes\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317800    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817789446 +0000 UTC m=+1.982191065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317817    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-etc-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317827    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-etc-kubernetes\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317827    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317834    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317864    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317874    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317879    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317903    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysconfig\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317910    4917 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317925    4917 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317929    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-ovn\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.317903 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.317946    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817938305 +0000 UTC m=+1.982339940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317962    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-socket-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317971    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-ovn\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317980    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysconfig\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.317990    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-k8s-cni-cncf-io\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.318005    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.817990368 +0000 UTC m=+1.982391994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : secret "metrics-daemon-secret" not found
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318009    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-socket-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318023    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-k8s-cni-cncf-io\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318036    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-kubelet\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318058    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-systemd-units\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318087    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-systemd-units\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318069    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-kubelet\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318095    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318122    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-conf-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318173    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-netd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.318176    4917 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318187    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-host\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318204    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-host-slash\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318217    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.318227    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.818215684 +0000 UTC m=+1.982617316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : configmap "kube-rbac-proxy" not found
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318237    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-run-openvswitch\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318178    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-multus-conf-dir\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318256    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-host\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318254    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318271    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-cni-netd\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318266    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4b078f-2e4a-4c6c-a352-16e2a2912bcb-host-slash\") pod \"iptables-alerter-d88b4\" (UID: \"1d4b078f-2e4a-4c6c-a352-16e2a2912bcb\") " pod="openshift-network-operator/iptables-alerter-d88b4"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318284    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-kubelet-config\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318299    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-slash\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318300    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318307    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-kubelet-config\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318313    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-registration-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318333    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-conf\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318336    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-host-slash\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318344    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d-registration-dir\") pod \"aws-ebs-csi-driver-node-xg5c7\" (UID: \"a2a32270-0ad2-4fb4-b15f-7dfcfb915a3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xg5c7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318347    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-var-lib-kubelet\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318363    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-dbus\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318378    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-log-socket\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318401    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-var-lib-kubelet\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318419    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-system-cni-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318426    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710b36be-68a9-4ed5-84c6-30d5e3dc6801-dbus\") pod \"global-pull-secret-syncer-x6kbk\" (UID: \"710b36be-68a9-4ed5-84c6-30d5e3dc6801\") " pod="kube-system/global-pull-secret-syncer-x6kbk"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318431    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d555bfc-7445-4727-a110-ce66dae3ee49-log-socket\") pod \"ovnkube-node-4qcq7\" (UID: \"4d555bfc-7445-4727-a110-ce66dae3ee49\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318431    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-conf\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318455    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-bin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318476    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-lib-modules\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318490    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49dc5027-2bf7-4365-a107-6dc6bf106382-host\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318506    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-netns\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318508    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f76c795-db38-4b17-b2d5-c3c3d9905edc-system-cni-dir\") pod \"multus-additional-cni-plugins-lngvb\" (UID: \"0f76c795-db38-4b17-b2d5-c3c3d9905edc\") " pod="openshift-multus/multus-additional-cni-plugins-lngvb"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318509    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-var-lib-cni-bin\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318531    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-sys\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318537    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49dc5027-2bf7-4365-a107-6dc6bf106382-host\") pod \"node-ca-t7wjp\" (UID: \"49dc5027-2bf7-4365-a107-6dc6bf106382\") " pod="openshift-image-registry/node-ca-t7wjp"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318554    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-hostroot\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318564    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-netns\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318568    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-lib-modules\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318579    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-multus-certs\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318583    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-sys\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318603    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318619    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-kubernetes\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318596    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-hostroot\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318633    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.318676    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.318687    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318687    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3-host-run-multus-certs\") pod \"multus-cn92p\" (UID: \"3ae9ccea-c7c3-4421-aad0-f36dd1cbc8e3\") " pod="openshift-multus/multus-cn92p"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318693    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-kubernetes\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.318711    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae767808-d262-4e49-8754-7149f5ec3fd1-etc-sysctl-d\") pod \"tuned-mrdzz\" (UID: \"ae767808-d262-4e49-8754-7149f5ec3fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-mrdzz"
Oct 30 19:41:45.319585 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.318720    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.818708944 +0000 UTC m=+1.983110586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:45.400720 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.400693    4917 scope.go:122] "RemoveContainer" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509"
Oct 30 19:41:45.412996 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.412978    4917 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:41:45.463310 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.463246    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:45.822500 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822429    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:45.822500 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822468    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:45.822500 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822488    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822507    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822573    4917 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822575    4917 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822604    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822614    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822603246 +0000 UTC m=+2.987004866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822577    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822634    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822644    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822629752 +0000 UTC m=+2.987031383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822654    4917 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822573    4917 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822666    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822656206 +0000 UTC m=+2.987057839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822686    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822675963 +0000 UTC m=+2.987077588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822709    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822703855 +0000 UTC m=+2.987105474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:45.822713 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822720    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822739    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822767    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822818    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822844    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822856    4917 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822864    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822874    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822885    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822869263 +0000 UTC m=+2.987270898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822863    4917 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822912    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822900777 +0000 UTC m=+2.987302400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822923    4917 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.822933    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822956    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822945231 +0000 UTC m=+2.987346897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : secret "metrics-daemon-secret" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822979    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822973226 +0000 UTC m=+2.987374846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:45.823212 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:45.822991    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:46.822985989 +0000 UTC m=+2.987387608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:45.823612 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:45.823352    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/292c4af8-ff18-45ab-96b2-bd05430793cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:46.051159 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.051131    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/2.log"
Oct 30 19:41:46.051534 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.051523    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/1.log"
Oct 30 19:41:46.051574 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.051552    4917 generic.go:356] "Generic (PLEG): container finished" podID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerID="73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a" exitCode=255
Oct 30 19:41:46.051659 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.051640    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerDied","Data":"73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a"}
Oct 30 19:41:46.051716 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.051685    4917 scope.go:122] "RemoveContainer" containerID="b92ef4c728138d5292076aa39ab1daa9fe5a0020cd7dc53c4b95fc7c29397509"
Oct 30 19:41:46.052468 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.052450    4917 scope.go:122] "RemoveContainer" containerID="73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a"
Oct 30 19:41:46.052619 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.052597    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:46.093808 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.093736    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:41:46.476638 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.476609    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:46.833141 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833056    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:46.833141 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833092    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:46.833141 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833119    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:46.833141 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833141    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833159    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833201    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833223    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833230    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833255    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833280    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833291    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833231    4917 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833312    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833294785 +0000 UTC m=+4.997696405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833323    4917 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833334    4917 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833338    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833328883 +0000 UTC m=+4.997730507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833237    4917 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833353    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833348181 +0000 UTC m=+4.997749801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833240    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833369    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833359013 +0000 UTC m=+4.997760634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833294    4917 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833391    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833384732 +0000 UTC m=+4.997786352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833405    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833398235 +0000 UTC m=+4.997799855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : secret "metrics-daemon-secret" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833427    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833415568 +0000 UTC m=+4.997817192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:46.833462 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833454    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.83344859 +0000 UTC m=+4.997850209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:46.834075 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833481    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:46.834075 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:46.833504    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:46.834075 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833565    4917 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:46.834075 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833568    4917 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:46.834075 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833586    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833579947 +0000 UTC m=+4.997981566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:46.834075 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:46.833595    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:48.833590499 +0000 UTC m=+4.997992118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:47.056395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:47.056367    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kpp2d" event={"ID":"b1efd70d-88c5-4c06-bea1-b998190bb385","Type":"ContainerStarted","Data":"152e9d3dd15e235a869c430a2ad3669e97d11cc1989986e064a52885421f73f6"}
Oct 30 19:41:47.057912 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:47.057893    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/2.log"
Oct 30 19:41:47.058079 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:47.058069    4917 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:41:48.038005 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.037952    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tznqp"
Oct 30 19:41:48.852236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852174    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:48.852236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852242    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852282    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852301    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852321    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852359    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852390    4917 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852395    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852423    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852449    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852433389 +0000 UTC m=+9.016835023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852447    4917 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852486    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852497    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852496    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852518    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852504809 +0000 UTC m=+9.016906446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852447    4917 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852542    4917 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852544    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852531879 +0000 UTC m=+9.016933514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852447    4917 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852570    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls podName:5de6c843-5c4d-4de8-99ff-950dd189b6bd nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852560466 +0000 UTC m=+9.016962086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls") pod "dns-default-9wckj" (UID: "5de6c843-5c4d-4de8-99ff-950dd189b6bd") : secret "dns-default-metrics-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852389    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852587    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852578808 +0000 UTC m=+9.016980433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : configmap references non-existent config key: service-ca.crt
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852603    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852595552 +0000 UTC m=+9.016997172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852627    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852612971 +0000 UTC m=+9.017014591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852645    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.8526395 +0000 UTC m=+9.017041119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852669    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:48.852690    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852734    4917 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852766    4917 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:41:48.852746 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852794    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852763701 +0000 UTC m=+9.017165336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:48.853583 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:48.852826    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:41:52.852819019 +0000 UTC m=+9.017220639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : secret "metrics-daemon-secret" not found
Oct 30 19:41:50.980822 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:50.980761    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:50.981313 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:50.981134    4917 scope.go:122] "RemoveContainer" containerID="73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a"
Oct 30 19:41:50.981313 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:50.981297    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:51.024568 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:51.024532    4917 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:51.075716 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:51.075690    4917 scope.go:122] "RemoveContainer" containerID="73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a"
Oct 30 19:41:51.075962 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:51.075931    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:52.572170 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.572142    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57p4t_56510de8-5bb1-45b5-a057-f83143d0d836/dns-node-resolver/0.log"
Oct 30 19:41:52.896078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.895989    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:41:52.896078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896032    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:41:52.896078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896065    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896094    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896120    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896143    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896165    4917 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896182    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896218    4917 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896224    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896260    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert podName:8e807294-3926-40b1-9203-9cbaf6f2ea40 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896240255 +0000 UTC m=+17.060641893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-csrgl" (UID: "8e807294-3926-40b1-9203-9cbaf6f2ea40") : secret "networking-console-plugin-cert" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896282    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896301    4917 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896305    4917 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896303    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b45499cc5-52dbl: secret "image-registry-tls" not found
Oct 30 19:41:52.896320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896287    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896305    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs podName:5b117693-4843-4840-a6f1-5567a04226e1 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896295155 +0000 UTC m=+17.060696776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs") pod "router-default-569cc6f5cf-pw4pl" (UID: "5b117693-4843-4840-a6f1-5567a04226e1") : secret "router-metrics-certs-default" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896354    4917 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896356    4917 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896370    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls podName:292c4af8-ff18-45ab-96b2-bd05430793cf nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896358221 +0000 UTC m=+17.060759842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwlvj" (UID: "292c4af8-ff18-45ab-96b2-bd05430793cf") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896384    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896377149 +0000 UTC m=+17.060778770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896397    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs podName:eb6f0ca7-6e21-4447-a036-2c91f24e885f nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896390876 +0000 UTC m=+17.060792497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs") pod "network-metrics-daemon-tfswf" (UID: "eb6f0ca7-6e21-4447-a036-2c91f24e885f") : secret "metrics-daemon-secret" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896416    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls podName:0932c21f-7a18-45f9-b244-c915394bf4d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896405307 +0000 UTC m=+17.060806927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls") pod "image-registry-7b45499cc5-52dbl" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6") : secret "image-registry-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896432    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls podName:bc01cb5a-688e-4b24-ab7b-8157db6ccc1f nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896427653 +0000 UTC m=+17.060829273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kk9wf" (UID: "bc01cb5a-688e-4b24-ab7b-8157db6ccc1f") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896520    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896531    4917 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896631    4917 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896647    4917 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d57c9fd74-7lvhr: secret "image-registry-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:52.896715    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls podName:ff11cf2e-b741-4ebc-a137-631442b49d23 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:00.896698667 +0000 UTC m=+17.061100305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls") pod "image-registry-6d57c9fd74-7lvhr" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23") : secret "image-registry-tls" not found
Oct 30 19:41:52.896906 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.896825    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b117693-4843-4840-a6f1-5567a04226e1-service-ca-bundle\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:41:52.898569 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.898554    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5de6c843-5c4d-4de8-99ff-950dd189b6bd-metrics-tls\") pod \"dns-default-9wckj\" (UID: \"5de6c843-5c4d-4de8-99ff-950dd189b6bd\") " pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:52.910436 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:52.910412    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:53.283918 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:53.283889    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9wckj"]
Oct 30 19:41:53.285954 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:41:53.285925    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de6c843_5c4d_4de8_99ff_950dd189b6bd.slice/crio-208cdd00211bf30077e41b1c24bc5a904cc5d6cb20348475d9d337e2b301ec94 WatchSource:0}: Error finding container 208cdd00211bf30077e41b1c24bc5a904cc5d6cb20348475d9d337e2b301ec94: Status 404 returned error can't find the container with id 208cdd00211bf30077e41b1c24bc5a904cc5d6cb20348475d9d337e2b301ec94
Oct 30 19:41:53.531184 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:53.531156    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qcq7"
Oct 30 19:41:53.903858 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:53.903828    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t7wjp_49dc5027-2bf7-4365-a107-6dc6bf106382/node-ca/0.log"
Oct 30 19:41:54.085483 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:54.085443    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wckj" event={"ID":"5de6c843-5c4d-4de8-99ff-950dd189b6bd","Type":"ContainerStarted","Data":"208cdd00211bf30077e41b1c24bc5a904cc5d6cb20348475d9d337e2b301ec94"}
Oct 30 19:41:54.302987 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:54.302957    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9n45f_4fe3fb9e-3883-4afe-ad11-64cd33045396/serve-healthcheck-canary/0.log"
Oct 30 19:41:54.605062 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:54.604992    4917 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-27.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:41:54.705929 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:54.705903    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-kpp2d_b1efd70d-88c5-4c06-bea1-b998190bb385/migrator/0.log"
Oct 30 19:41:55.110680 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:55.110651    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-rh65c_31ae3d8c-faa0-468e-a557-d9f870fad555/kube-storage-version-migrator-operator/0.log"
Oct 30 19:41:56.093732 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:56.093656    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wckj" event={"ID":"5de6c843-5c4d-4de8-99ff-950dd189b6bd","Type":"ContainerStarted","Data":"740a322e79df5ed738251e6a285277b50a06b36d7798aa6e0441b8b18fc58300"}
Oct 30 19:41:56.093732 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:56.093707    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wckj" event={"ID":"5de6c843-5c4d-4de8-99ff-950dd189b6bd","Type":"ContainerStarted","Data":"c8e0ac9c4872bee1779d6de4dbb263807afa9267b30a01b613ce570248224019"}
Oct 30 19:41:56.093920 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:56.093810    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9wckj"
Oct 30 19:41:56.995217 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:56.995187    4917 scope.go:122] "RemoveContainer" containerID="73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a"
Oct 30 19:41:57.100675 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:57.100503    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/2.log"
Oct 30 19:41:57.100675 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:57.100597    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerStarted","Data":"356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97"}
Oct 30 19:41:57.101226 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:57.101200    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:41:57.102810 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:57.102783    4917 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pjlwh container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Oct 30 19:41:57.102911 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:57.102855    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused"
Oct 30 19:41:58.109102 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:58.109071    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:41:58.109514 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:58.109458    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/2.log"
Oct 30 19:41:58.109514 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:58.109505    4917 generic.go:356] "Generic (PLEG): container finished" podID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerID="356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97" exitCode=255
Oct 30 19:41:58.109588 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:58.109543    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerDied","Data":"356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97"}
Oct 30 19:41:58.109632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:58.109619    4917 scope.go:122] "RemoveContainer" containerID="73c16256bf438b974878c341ff4935458e171900df96cd55041e3266c22c6c7a"
Oct 30 19:41:58.109816 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:58.109801    4917 scope.go:122] "RemoveContainer" containerID="356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97"
Oct 30 19:41:58.110010 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:58.109992    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:41:59.113715 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:59.113683    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:41:59.114112 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:41:59.114015    4917 scope.go:122] "RemoveContainer" containerID="356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97"
Oct 30 19:41:59.114202 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:41:59.114184    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:42:00.969415 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969373    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969425    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969443    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969468    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969503    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969527    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969550    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.969564    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:00.969699    4917 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:42:00.969962 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:00.969802    4917 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls podName:ef755a60-5e9e-4ba7-9305-e3efbc283bf6 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:16.969763047 +0000 UTC m=+33.134164667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rrgxl" (UID: "ef755a60-5e9e-4ba7-9305-e3efbc283bf6") : secret "samples-operator-tls" not found
Oct 30 19:42:00.972417 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.972387    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"image-registry-7b45499cc5-52dbl\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") " pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:00.972529 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.972513    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b117693-4843-4840-a6f1-5567a04226e1-metrics-certs\") pod \"router-default-569cc6f5cf-pw4pl\" (UID: \"5b117693-4843-4840-a6f1-5567a04226e1\") " pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:00.972576 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.972554    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/292c4af8-ff18-45ab-96b2-bd05430793cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwlvj\" (UID: \"292c4af8-ff18-45ab-96b2-bd05430793cf\") " pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:42:00.972647 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.972629    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"image-registry-6d57c9fd74-7lvhr\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") " pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:00.972860 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.972840    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e807294-3926-40b1-9203-9cbaf6f2ea40-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-csrgl\" (UID: \"8e807294-3926-40b1-9203-9cbaf6f2ea40\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:42:00.972986 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.972967    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f0ca7-6e21-4447-a036-2c91f24e885f-metrics-certs\") pod \"network-metrics-daemon-tfswf\" (UID: \"eb6f0ca7-6e21-4447-a036-2c91f24e885f\") " pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:42:00.973040 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.973026    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc01cb5a-688e-4b24-ab7b-8157db6ccc1f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kk9wf\" (UID: \"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:42:00.998250 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:00.998229    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:01.000317 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.000299    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"
Oct 30 19:42:01.000395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.000373    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:01.001871 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.001855    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl"
Oct 30 19:42:01.003082 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.003066    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfswf"
Oct 30 19:42:01.008317 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.008293    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:01.014966 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.014946    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwlvj"
Oct 30 19:42:01.025571 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.025548    4917 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:42:01.026047 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.025988    4917 scope.go:122] "RemoveContainer" containerID="356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97"
Oct 30 19:42:01.028311 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:01.028264    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pjlwh_openshift-console-operator(9e23665b-8521-44d0-b403-ff1fefae7d74)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74"
Oct 30 19:42:01.523063 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:42:01.523034    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b117693_4843_4840_a6f1_5567a04226e1.slice/crio-fdec73a79a870fd3ec99bce514e9d7d8e3c7402444ab2473e99943e130f65e44 WatchSource:0}: Error finding container fdec73a79a870fd3ec99bce514e9d7d8e3c7402444ab2473e99943e130f65e44: Status 404 returned error can't find the container with id fdec73a79a870fd3ec99bce514e9d7d8e3c7402444ab2473e99943e130f65e44
Oct 30 19:42:01.523708 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:42:01.523671    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff11cf2e_b741_4ebc_a137_631442b49d23.slice/crio-88248133398cc9355a0086a5325b8eabaf595f8e3e1fd4490cc456de1307c901 WatchSource:0}: Error finding container 88248133398cc9355a0086a5325b8eabaf595f8e3e1fd4490cc456de1307c901: Status 404 returned error can't find the container with id 88248133398cc9355a0086a5325b8eabaf595f8e3e1fd4490cc456de1307c901
Oct 30 19:42:01.564287 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.564244    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-569cc6f5cf-pw4pl"]
Oct 30 19:42:01.571229 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.569526    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d57c9fd74-7lvhr"]
Oct 30 19:42:01.571229 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.570523    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-csrgl"]
Oct 30 19:42:01.573142 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:42:01.573104    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e807294_3926_40b1_9203_9cbaf6f2ea40.slice/crio-1f79850bc76e528d5edc9a59a97d479208a582b8b3ff20195a480cb16eb8651c WatchSource:0}: Error finding container 1f79850bc76e528d5edc9a59a97d479208a582b8b3ff20195a480cb16eb8651c: Status 404 returned error can't find the container with id 1f79850bc76e528d5edc9a59a97d479208a582b8b3ff20195a480cb16eb8651c
Oct 30 19:42:01.574158 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:42:01.573939    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292c4af8_ff18_45ab_96b2_bd05430793cf.slice/crio-a29a8e4dc220204fe12614ebbc09da3d356c5118f76c75800c8ee37da1a1f8c4 WatchSource:0}: Error finding container a29a8e4dc220204fe12614ebbc09da3d356c5118f76c75800c8ee37da1a1f8c4: Status 404 returned error can't find the container with id a29a8e4dc220204fe12614ebbc09da3d356c5118f76c75800c8ee37da1a1f8c4
Oct 30 19:42:01.578424 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.577715    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tfswf"]
Oct 30 19:42:01.578712 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:42:01.578685    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0932c21f_7a18_45f9_b244_c915394bf4d6.slice/crio-e43197e0040100e2cca1ff753f0af877ba6de10f20de6327424598cf395164d7 WatchSource:0}: Error finding container e43197e0040100e2cca1ff753f0af877ba6de10f20de6327424598cf395164d7: Status 404 returned error can't find the container with id e43197e0040100e2cca1ff753f0af877ba6de10f20de6327424598cf395164d7
Oct 30 19:42:01.615803 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.615357    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf"]
Oct 30 19:42:01.621025 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.620967    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwlvj"]
Oct 30 19:42:01.621126 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:01.621041    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b45499cc5-52dbl"]
Oct 30 19:42:02.130142 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.130101    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwlvj" event={"ID":"292c4af8-ff18-45ab-96b2-bd05430793cf","Type":"ContainerStarted","Data":"4a5d58d5e67404db886254c5faedbb4c8dcb02e0cf606c8913129c933a5d34a7"}
Oct 30 19:42:02.130581 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.130162    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwlvj" event={"ID":"292c4af8-ff18-45ab-96b2-bd05430793cf","Type":"ContainerStarted","Data":"a29a8e4dc220204fe12614ebbc09da3d356c5118f76c75800c8ee37da1a1f8c4"}
Oct 30 19:42:02.131911 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.131878    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl" event={"ID":"8e807294-3926-40b1-9203-9cbaf6f2ea40","Type":"ContainerStarted","Data":"1f79850bc76e528d5edc9a59a97d479208a582b8b3ff20195a480cb16eb8651c"}
Oct 30 19:42:02.133023 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.132999    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfswf" event={"ID":"eb6f0ca7-6e21-4447-a036-2c91f24e885f","Type":"ContainerStarted","Data":"55c98922c4c12f132b9b48cc63683e97bcd5596ae0f8ede7780819313da93edb"}
Oct 30 19:42:02.134453 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.134423    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl" event={"ID":"5b117693-4843-4840-a6f1-5567a04226e1","Type":"ContainerStarted","Data":"82c002d61fa426ab1573809cfd15e297c4faacda82511479f6a6f6f12b7ecfbe"}
Oct 30 19:42:02.134690 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.134464    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl" event={"ID":"5b117693-4843-4840-a6f1-5567a04226e1","Type":"ContainerStarted","Data":"fdec73a79a870fd3ec99bce514e9d7d8e3c7402444ab2473e99943e130f65e44"}
Oct 30 19:42:02.135848 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.135826    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl" event={"ID":"0932c21f-7a18-45f9-b244-c915394bf4d6","Type":"ContainerStarted","Data":"452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad"}
Oct 30 19:42:02.135943 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.135858    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl" event={"ID":"0932c21f-7a18-45f9-b244-c915394bf4d6","Type":"ContainerStarted","Data":"e43197e0040100e2cca1ff753f0af877ba6de10f20de6327424598cf395164d7"}
Oct 30 19:42:02.135943 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.135904    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:02.137315 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.137296    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr" event={"ID":"ff11cf2e-b741-4ebc-a137-631442b49d23","Type":"ContainerStarted","Data":"bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313"}
Oct 30 19:42:02.137382 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.137326    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr" event={"ID":"ff11cf2e-b741-4ebc-a137-631442b49d23","Type":"ContainerStarted","Data":"88248133398cc9355a0086a5325b8eabaf595f8e3e1fd4490cc456de1307c901"}
Oct 30 19:42:02.137382 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.137365    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:02.138461 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:02.138443    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf" event={"ID":"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f","Type":"ContainerStarted","Data":"799ab884ac59fa78f8154fc52caf5523acb7d4f0897e1c0ed477ab3e34bcf284"}
Oct 30 19:42:03.001387 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:03.001356    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:03.004413 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:03.004380    4917 patch_prober.go:29] interesting pod/router-default-569cc6f5cf-pw4pl container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 30 19:42:03.004413 ip-10-0-133-27 kubenswrapper[4917]: [-]has-synced failed: reason withheld
Oct 30 19:42:03.004413 ip-10-0-133-27 kubenswrapper[4917]: [+]process-running ok
Oct 30 19:42:03.004413 ip-10-0-133-27 kubenswrapper[4917]: healthz check failed
Oct 30 19:42:03.004651 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:03.004438    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl" podUID="5b117693-4843-4840-a6f1-5567a04226e1" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 30 19:42:03.146297 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:03.146185    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwlvj" event={"ID":"292c4af8-ff18-45ab-96b2-bd05430793cf","Type":"ContainerStarted","Data":"c5db91299f7b01950a50f1835a7e1c2763610ba41e6929878cd92373d2ba9d82"}
Oct 30 19:42:04.002886 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:04.002852    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:04.165471 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:04.165431    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-csrgl" event={"ID":"8e807294-3926-40b1-9203-9cbaf6f2ea40","Type":"ContainerStarted","Data":"9c5c9998d2ebc38da4545d75f36c29c653d280dadeec96617a6584283ba60588"}
Oct 30 19:42:04.167847 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:04.167636    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfswf" event={"ID":"eb6f0ca7-6e21-4447-a036-2c91f24e885f","Type":"ContainerStarted","Data":"dca369f4a863e7335a29cdf04a22e12c7ce27a615ebce20000c8ee3bb04af6a7"}
Oct 30 19:42:04.168106 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:04.168083    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:04.168216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:04.168121    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfswf" event={"ID":"eb6f0ca7-6e21-4447-a036-2c91f24e885f","Type":"ContainerStarted","Data":"3fd010e00d671e90ed72060bca3d2d0b6e358230de5948b2e87bd5fd940bc1cf"}
Oct 30 19:42:04.169164 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:04.169145    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-569cc6f5cf-pw4pl"
Oct 30 19:42:05.172969 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:05.172884    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kk9wf" event={"ID":"bc01cb5a-688e-4b24-ab7b-8157db6ccc1f","Type":"ContainerStarted","Data":"18a11f5cafef039cce31bdc938a26a95bf8eb11c418bc4e6e7f07c0f94bfb70f"}
Oct 30 19:42:05.174998 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:05.174970    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwlvj" event={"ID":"292c4af8-ff18-45ab-96b2-bd05430793cf","Type":"ContainerStarted","Data":"505b8212632e18a4172bc6e5a6898eab9e2fb9aaa3ab51653da335d5d93d124c"}
Oct 30 19:42:06.103286 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:06.103255    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9wckj"
Oct 30 19:42:09.677906 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:09.677856    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:42:11.415170 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:11.415143    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d57c9fd74-7lvhr"]
Oct 30 19:42:16.096370 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:16.096323    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pt4mt"
Oct 30 19:42:17.002068 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:17.002029    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:42:17.004444 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:17.004413    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef755a60-5e9e-4ba7-9305-e3efbc283bf6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rrgxl\" (UID: \"ef755a60-5e9e-4ba7-9305-e3efbc283bf6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:42:17.199515 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:17.199482    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"
Oct 30 19:42:17.671742 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:17.671657    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl"]
Oct 30 19:42:18.221933 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:18.221894    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl" event={"ID":"ef755a60-5e9e-4ba7-9305-e3efbc283bf6","Type":"ContainerStarted","Data":"a899f28e51c32fac08af066820ac1d9e1c7e77e0de228dc83ead5db33cb2412f"}
Oct 30 19:42:20.231398 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:20.231368    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl" event={"ID":"ef755a60-5e9e-4ba7-9305-e3efbc283bf6","Type":"ContainerStarted","Data":"d4e4d0084aec5c9574c16f7dafb6d30c8c5ad9661c7ad84992bcadbc12b578af"}
Oct 30 19:42:21.236745 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:21.236710    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rrgxl" event={"ID":"ef755a60-5e9e-4ba7-9305-e3efbc283bf6","Type":"ContainerStarted","Data":"f62ab291a21c96760e71c649a19af0e1bf449288c36f879b63dd6e127ca09963"}
Oct 30 19:42:21.420632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:21.420599    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:23.151237 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.151188    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:23.190493 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.190461    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lmdpj"]
Oct 30 19:42:23.192661 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.192639    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.251898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.251868    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-tls\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.251898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.251899    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0b293ce-2016-4abb-ae21-9e88058f087f-metrics-client-ca\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252079 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.251939    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-sys\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252146 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.252110    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-wtmp\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252266 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.252205    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-textfile\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252266 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.252260    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-root\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252365 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.252337    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljtq\" (UniqueName: \"kubernetes.io/projected/a0b293ce-2016-4abb-ae21-9e88058f087f-kube-api-access-6ljtq\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252400 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.252381    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-accelerators-collector-config\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.252432 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.252407    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.289975 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.289950    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:42:23.290098 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.290017    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:42:23.305428 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.305408    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:42:23.305884 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.305872    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:42:23.306116 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.306103    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fn2xq\""
Oct 30 19:42:23.352782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352735    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-wtmp\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352790    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-textfile\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352833    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-root\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352856    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljtq\" (UniqueName: \"kubernetes.io/projected/a0b293ce-2016-4abb-ae21-9e88058f087f-kube-api-access-6ljtq\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352885    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-accelerators-collector-config\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352911    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352926    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-wtmp\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.352948 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352940    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-tls\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353305 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.352965    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0b293ce-2016-4abb-ae21-9e88058f087f-metrics-client-ca\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353305 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.353013    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-sys\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353305 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.353070    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-root\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353305 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.353155    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0b293ce-2016-4abb-ae21-9e88058f087f-sys\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353305 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.353214    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-textfile\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353560 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.353518    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-accelerators-collector-config\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.353687 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.353666    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0b293ce-2016-4abb-ae21-9e88058f087f-metrics-client-ca\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.355751 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.355727    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.355860 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.355764    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0b293ce-2016-4abb-ae21-9e88058f087f-node-exporter-tls\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.415530 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.415468    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljtq\" (UniqueName: \"kubernetes.io/projected/a0b293ce-2016-4abb-ae21-9e88058f087f-kube-api-access-6ljtq\") pod \"node-exporter-lmdpj\" (UID: \"a0b293ce-2016-4abb-ae21-9e88058f087f\") " pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.505172 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.505139    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lmdpj"
Oct 30 19:42:23.515167 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:42:23.515134    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b293ce_2016_4abb_ae21_9e88058f087f.slice/crio-4fdb91989890b12f472c84d2d261dcb63f002710359d0ffadf6f85300b45cafe WatchSource:0}: Error finding container 4fdb91989890b12f472c84d2d261dcb63f002710359d0ffadf6f85300b45cafe: Status 404 returned error can't find the container with id 4fdb91989890b12f472c84d2d261dcb63f002710359d0ffadf6f85300b45cafe
Oct 30 19:42:23.996517 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:23.996430    4917 scope.go:122] "RemoveContainer" containerID="356f04699355d923f5b6d6b048d0ea3e49b3521b876652fd6d053527a8864b97"
Oct 30 19:42:24.249170 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:24.249081    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmdpj" event={"ID":"a0b293ce-2016-4abb-ae21-9e88058f087f","Type":"ContainerStarted","Data":"4fdb91989890b12f472c84d2d261dcb63f002710359d0ffadf6f85300b45cafe"}
Oct 30 19:42:24.250947 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:24.250925    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:42:24.251069 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:24.250968    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" event={"ID":"9e23665b-8521-44d0-b403-ff1fefae7d74","Type":"ContainerStarted","Data":"4e4ce715d9432d87361ec263a0c287993465f3cfd519295826a4a6d51d4ebaa0"}
Oct 30 19:42:24.251234 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:24.251215    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:42:24.252219 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:24.252196    4917 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pjlwh container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Oct 30 19:42:24.252328 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:24.252235    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh" podUID="9e23665b-8521-44d0-b403-ff1fefae7d74" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused"
Oct 30 19:42:25.257154 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:25.257117    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmdpj" event={"ID":"a0b293ce-2016-4abb-ae21-9e88058f087f","Type":"ContainerStarted","Data":"0aab57e230c76b49c9a5b7ca565dfd6df088949dbc9249d95d16d3c5c1e71a69"}
Oct 30 19:42:25.262040 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:25.262019    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pjlwh"
Oct 30 19:42:26.263472 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:26.263435    4917 generic.go:356] "Generic (PLEG): container finished" podID="a0b293ce-2016-4abb-ae21-9e88058f087f" containerID="0aab57e230c76b49c9a5b7ca565dfd6df088949dbc9249d95d16d3c5c1e71a69" exitCode=0
Oct 30 19:42:26.264019 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:26.263497    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmdpj" event={"ID":"a0b293ce-2016-4abb-ae21-9e88058f087f","Type":"ContainerDied","Data":"0aab57e230c76b49c9a5b7ca565dfd6df088949dbc9249d95d16d3c5c1e71a69"}
Oct 30 19:42:26.264019 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:26.263563    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmdpj" event={"ID":"a0b293ce-2016-4abb-ae21-9e88058f087f","Type":"ContainerStarted","Data":"d4244e1dc1e479a31e86de908d9fde8802a97a8b4e46c6c9ffdd5ff19494a082"}
Oct 30 19:42:26.264019 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:26.263591    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lmdpj" event={"ID":"a0b293ce-2016-4abb-ae21-9e88058f087f","Type":"ContainerStarted","Data":"676c4386d14c80d2a1fda2ea81dbe73cc4fd0822d50f092dc5818d4d1d3ccd86"}
Oct 30 19:42:26.418460 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:26.418406    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lmdpj" podStartSLOduration=1.756328646 podStartE2EDuration="3.418392392s" totalImagesPullingTime="1.592712873s" totalInitContainerRuntime="69.350873ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:26.417884463 +0000 UTC m=+42.582286104" watchObservedRunningTime="2026-10-30 19:42:26.418392392 +0000 UTC m=+42.582794034"
Oct 30 19:42:33.687446 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:33.687409    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b45499cc5-52dbl"]
Oct 30 19:42:36.434597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.434540    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr" podUID="ff11cf2e-b741-4ebc-a137-631442b49d23" containerName="registry" containerID="cri-o://bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313" gracePeriod=30
Oct 30 19:42:36.673167 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.673141    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:36.776686 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776620    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.776686 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776652    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.776686 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776680    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.777011 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776728    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.777011 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776795    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.777011 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776831    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.777011 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776879    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.777011 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.776954    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") pod \"ff11cf2e-b741-4ebc-a137-631442b49d23\" (UID: \"ff11cf2e-b741-4ebc-a137-631442b49d23\") "
Oct 30 19:42:36.777277 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.777192    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:36.777547 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.777498    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:36.779456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.779425    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:36.779572 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.779458    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:36.779572 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.779431    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:36.779572 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.779543    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:36.779572 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.779567    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "kube-api-access-54pgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:36.783940 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.783921    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted" pod "ff11cf2e-b741-4ebc-a137-631442b49d23" (UID: "ff11cf2e-b741-4ebc-a137-631442b49d23"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:36.878321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878283    4917 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-image-registry-private-configuration\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878321    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54pgw\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-kube-api-access-54pgw\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878340    4917 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-bound-sa-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878357    4917 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-tls\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878372    4917 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-trusted-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878386    4917 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff11cf2e-b741-4ebc-a137-631442b49d23-ca-trust-extracted\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878402    4917 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff11cf2e-b741-4ebc-a137-631442b49d23-registry-certificates\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:36.878516 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:36.878417    4917 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff11cf2e-b741-4ebc-a137-631442b49d23-installation-pull-secrets\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:37.307581 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.307546    4917 generic.go:356] "Generic (PLEG): container finished" podID="ff11cf2e-b741-4ebc-a137-631442b49d23" containerID="bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313" exitCode=0
Oct 30 19:42:37.307749 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.307591    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr"
Oct 30 19:42:37.307749 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.307606    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr" event={"ID":"ff11cf2e-b741-4ebc-a137-631442b49d23","Type":"ContainerDied","Data":"bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313"}
Oct 30 19:42:37.307749 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.307642    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d57c9fd74-7lvhr" event={"ID":"ff11cf2e-b741-4ebc-a137-631442b49d23","Type":"ContainerDied","Data":"88248133398cc9355a0086a5325b8eabaf595f8e3e1fd4490cc456de1307c901"}
Oct 30 19:42:37.307749 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.307666    4917 scope.go:122] "RemoveContainer" containerID="bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313"
Oct 30 19:42:37.315999 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.315980    4917 scope.go:122] "RemoveContainer" containerID="bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313"
Oct 30 19:42:37.316284 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:37.316264    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313\": container with ID starting with bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313 not found: ID does not exist" containerID="bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313"
Oct 30 19:42:37.316348 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.316294    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313"} err="failed to get container status \"bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313\": rpc error: code = NotFound desc = could not find container \"bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313\": container with ID starting with bde7499e4b8c21b0783a250ff69d304867b2f21a9aec4f48d25307c1f57e3313 not found: ID does not exist"
Oct 30 19:42:37.483077 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.483046    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d57c9fd74-7lvhr"]
Oct 30 19:42:37.533668 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:37.533629    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d57c9fd74-7lvhr"]
Oct 30 19:42:38.004007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:38.001281    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff11cf2e-b741-4ebc-a137-631442b49d23" path="/var/lib/kubelet/pods/ff11cf2e-b741-4ebc-a137-631442b49d23/volumes"
Oct 30 19:42:39.679354 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:39.679315    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:42:54.370297 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:54.370264    4917 generic.go:356] "Generic (PLEG): container finished" podID="31ae3d8c-faa0-468e-a557-d9f870fad555" containerID="d645aeff79c60a051076a93950491b26a79bbd8cba5994c0a79ed3f5d0ef6ff2" exitCode=0
Oct 30 19:42:54.370720 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:54.370339    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c" event={"ID":"31ae3d8c-faa0-468e-a557-d9f870fad555","Type":"ContainerDied","Data":"d645aeff79c60a051076a93950491b26a79bbd8cba5994c0a79ed3f5d0ef6ff2"}
Oct 30 19:42:54.370810 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:54.370741    4917 scope.go:122] "RemoveContainer" containerID="d645aeff79c60a051076a93950491b26a79bbd8cba5994c0a79ed3f5d0ef6ff2"
Oct 30 19:42:55.375849 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:55.375812    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rh65c" event={"ID":"31ae3d8c-faa0-468e-a557-d9f870fad555","Type":"ContainerStarted","Data":"89eb5a08c2343d2f2af567382f1bdabb7ac29f62f77a7d3bf04d3ebb7762a61b"}
Oct 30 19:42:58.708228 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:58.708186    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl" podUID="0932c21f-7a18-45f9-b244-c915394bf4d6" containerName="registry" containerID="cri-o://452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad" gracePeriod=30
Oct 30 19:42:58.959682 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:58.959621    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:59.150353 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150316    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150374    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150407    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150443    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150463    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150491    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150540 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150520    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150889 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150555    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") pod \"0932c21f-7a18-45f9-b244-c915394bf4d6\" (UID: \"0932c21f-7a18-45f9-b244-c915394bf4d6\") "
Oct 30 19:42:59.150982 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.150814    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:59.151314 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.151257    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:59.153532 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.153501    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:59.153633 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.153565    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:59.153633 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.153580    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:59.153633 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.153591    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:59.153734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.153662    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "kube-api-access-lp6ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:59.157687 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.157645    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted" pod "0932c21f-7a18-45f9-b244-c915394bf4d6" (UID: "0932c21f-7a18-45f9-b244-c915394bf4d6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:59.251456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251364    4917 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-certificates\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251399    4917 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-image-registry-private-configuration\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251413    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp6ff\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-kube-api-access-lp6ff\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251427    4917 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-bound-sa-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251439    4917 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0932c21f-7a18-45f9-b244-c915394bf4d6-trusted-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251452    4917 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0932c21f-7a18-45f9-b244-c915394bf4d6-ca-trust-extracted\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251750 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251464    4917 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0932c21f-7a18-45f9-b244-c915394bf4d6-installation-pull-secrets\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.251750 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.251477    4917 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0932c21f-7a18-45f9-b244-c915394bf4d6-registry-tls\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.392347 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.392313    4917 generic.go:356] "Generic (PLEG): container finished" podID="0932c21f-7a18-45f9-b244-c915394bf4d6" containerID="452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad" exitCode=0
Oct 30 19:42:59.392347 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.392364    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl"
Oct 30 19:42:59.392597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.392402    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl" event={"ID":"0932c21f-7a18-45f9-b244-c915394bf4d6","Type":"ContainerDied","Data":"452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad"}
Oct 30 19:42:59.392597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.392454    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b45499cc5-52dbl" event={"ID":"0932c21f-7a18-45f9-b244-c915394bf4d6","Type":"ContainerDied","Data":"e43197e0040100e2cca1ff753f0af877ba6de10f20de6327424598cf395164d7"}
Oct 30 19:42:59.392597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.392480    4917 scope.go:122] "RemoveContainer" containerID="452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad"
Oct 30 19:42:59.400740 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.400721    4917 scope.go:122] "RemoveContainer" containerID="452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad"
Oct 30 19:42:59.401041 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:42:59.401021    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad\": container with ID starting with 452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad not found: ID does not exist" containerID="452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad"
Oct 30 19:42:59.401140 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.401047    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad"} err="failed to get container status \"452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad\": rpc error: code = NotFound desc = could not find container \"452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad\": container with ID starting with 452f36aa5334279145f11c3c2db3b00aaca18478fbbd4975483277a6624153ad not found: ID does not exist"
Oct 30 19:42:59.567370 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.567285    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b45499cc5-52dbl"]
Oct 30 19:42:59.569482 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:42:59.569464    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b45499cc5-52dbl"]
Oct 30 19:43:00.000451 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:00.000418    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0932c21f-7a18-45f9-b244-c915394bf4d6" path="/var/lib/kubelet/pods/0932c21f-7a18-45f9-b244-c915394bf4d6/volumes"
Oct 30 19:43:01.401268 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:01.401234    4917 generic.go:356] "Generic (PLEG): container finished" podID="a0070d9d-efde-4889-87e9-39b47aefbe58" containerID="7b6e6adeb29f7437ca78d3df49e96b04290b5b3ec99e7779d8bf65ca9b1e35f8" exitCode=0
Oct 30 19:43:01.401629 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:01.401309    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv" event={"ID":"a0070d9d-efde-4889-87e9-39b47aefbe58","Type":"ContainerDied","Data":"7b6e6adeb29f7437ca78d3df49e96b04290b5b3ec99e7779d8bf65ca9b1e35f8"}
Oct 30 19:43:01.401679 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:01.401666    4917 scope.go:122] "RemoveContainer" containerID="7b6e6adeb29f7437ca78d3df49e96b04290b5b3ec99e7779d8bf65ca9b1e35f8"
Oct 30 19:43:02.407427 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:02.407394    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-56vpv" event={"ID":"a0070d9d-efde-4889-87e9-39b47aefbe58","Type":"ContainerStarted","Data":"2cfe05d61ff08195eedc560c85681e4f7a6740ccb33a000cd9f33e4907622e9b"}
Oct 30 19:43:09.678577 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:43:09.678541    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:43:10.440499 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:10.440467    4917 generic.go:356] "Generic (PLEG): container finished" podID="83293a26-814d-4b38-bf8e-10b28947c6c2" containerID="d91726df0ef2995803cc01962f8e664712cc93933b2e508ae36048377eac691e" exitCode=0
Oct 30 19:43:10.440678 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:10.440541    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t9qb9" event={"ID":"83293a26-814d-4b38-bf8e-10b28947c6c2","Type":"ContainerDied","Data":"d91726df0ef2995803cc01962f8e664712cc93933b2e508ae36048377eac691e"}
Oct 30 19:43:10.440914 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:10.440900    4917 scope.go:122] "RemoveContainer" containerID="d91726df0ef2995803cc01962f8e664712cc93933b2e508ae36048377eac691e"
Oct 30 19:43:11.448216 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:43:11.448176    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-t9qb9" event={"ID":"83293a26-814d-4b38-bf8e-10b28947c6c2","Type":"ContainerStarted","Data":"f892aba01d60ebfb328bf90d181ae537b319771477ba7b56f6948af3749e4c54"}
Oct 30 19:43:39.678751 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:43:39.678713    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:44:09.677484 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:44:09.677437    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:44:39.678640 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:44:39.678597    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:45:09.678240 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:45:09.678201    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:45:39.678573 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:45:39.678490    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:46:09.677342 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:46:09.677300    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:46:39.678486 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:46:39.678448    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:46:43.998689 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:46:43.998656    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:46:43.999161 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:46:43.999096    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:47:09.677725 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:47:09.677687    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:47:31.907107 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907070    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"]
Oct 30 19:47:31.907583 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907366    4917 cpu_manager.go:414] "containerMap: removing container" podUID="0932c21f-7a18-45f9-b244-c915394bf4d6" containerName="registry"
Oct 30 19:47:31.907583 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907377    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0932c21f-7a18-45f9-b244-c915394bf4d6" containerName="registry"
Oct 30 19:47:31.907583 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907393    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ff11cf2e-b741-4ebc-a137-631442b49d23" containerName="registry"
Oct 30 19:47:31.907583 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907401    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff11cf2e-b741-4ebc-a137-631442b49d23" containerName="registry"
Oct 30 19:47:31.907583 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907466    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="0932c21f-7a18-45f9-b244-c915394bf4d6" containerName="registry"
Oct 30 19:47:31.907583 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.907474    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff11cf2e-b741-4ebc-a137-631442b49d23" containerName="registry"
Oct 30 19:47:31.910503 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.910485    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:31.973053 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.973027    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:31.973177 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.973076    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmqc\" (UniqueName: \"kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:31.973177 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:31.973151    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.014422 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.014393    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:47:32.014422 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.014421    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-w2m75\""
Oct 30 19:47:32.014613 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.014595    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:47:32.015616 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.015596    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"]
Oct 30 19:47:32.074461 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.074433    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.074623 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.074495    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmqc\" (UniqueName: \"kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.074623 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.074536    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.074831 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.074813    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.074939 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.074921    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.138283 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.138254    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmqc\" (UniqueName: \"kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.226065 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.226040    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:32.596782 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.596688    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"]
Oct 30 19:47:32.599617 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:47:32.599585    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825d9b60_892f_4110_b8b9_ebd26a38f4a3.slice/crio-3a8f02697e59a2bee9308ec3eddca7789511007199d21d65e67e97db1e8139c8 WatchSource:0}: Error finding container 3a8f02697e59a2bee9308ec3eddca7789511007199d21d65e67e97db1e8139c8: Status 404 returned error can't find the container with id 3a8f02697e59a2bee9308ec3eddca7789511007199d21d65e67e97db1e8139c8
Oct 30 19:47:32.601555 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:32.601529    4917 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:47:33.351343 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:33.351308    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" event={"ID":"825d9b60-892f-4110-b8b9-ebd26a38f4a3","Type":"ContainerStarted","Data":"3a8f02697e59a2bee9308ec3eddca7789511007199d21d65e67e97db1e8139c8"}
Oct 30 19:47:38.377480 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:38.377441    4917 generic.go:356] "Generic (PLEG): container finished" podID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerID="6bc777cfe4b5282fd9bacabf1320e6080cb70d0b9a647545f67d13804cf8470c" exitCode=0
Oct 30 19:47:38.377841 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:38.377517    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" event={"ID":"825d9b60-892f-4110-b8b9-ebd26a38f4a3","Type":"ContainerDied","Data":"6bc777cfe4b5282fd9bacabf1320e6080cb70d0b9a647545f67d13804cf8470c"}
Oct 30 19:47:39.678316 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:47:39.678280    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:47:41.391206 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:41.391177    4917 generic.go:356] "Generic (PLEG): container finished" podID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerID="ed8b3ff96bac70664507c8abbd4eeec1f1e702be669182fc66932b360906eeaa" exitCode=0
Oct 30 19:47:41.391608 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:41.391230    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" event={"ID":"825d9b60-892f-4110-b8b9-ebd26a38f4a3","Type":"ContainerDied","Data":"ed8b3ff96bac70664507c8abbd4eeec1f1e702be669182fc66932b360906eeaa"}
Oct 30 19:47:48.423764 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:48.423728    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" event={"ID":"825d9b60-892f-4110-b8b9-ebd26a38f4a3","Type":"ContainerStarted","Data":"39c536e7c225cc9ee7edb8853e7df1b5e12afbdfdae1cbf62a5682843ebda2a3"}
Oct 30 19:47:48.544433 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:48.544366    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" podStartSLOduration=2.6945710910000003 podStartE2EDuration="17.544346174s" totalImagesPullingTime="14.849775083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:48.543098221 +0000 UTC m=+364.707499863" watchObservedRunningTime="2026-10-30 19:47:48.544346174 +0000 UTC m=+364.708747828"
Oct 30 19:47:49.429695 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:49.429657    4917 generic.go:356] "Generic (PLEG): container finished" podID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerID="39c536e7c225cc9ee7edb8853e7df1b5e12afbdfdae1cbf62a5682843ebda2a3" exitCode=0
Oct 30 19:47:49.430125 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:49.429747    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" event={"ID":"825d9b60-892f-4110-b8b9-ebd26a38f4a3","Type":"ContainerDied","Data":"39c536e7c225cc9ee7edb8853e7df1b5e12afbdfdae1cbf62a5682843ebda2a3"}
Oct 30 19:47:50.556191 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.556164    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:50.628142 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.628112    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util\") pod \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") "
Oct 30 19:47:50.628294 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.628147    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc\" (UniqueName: \"kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc\") pod \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") "
Oct 30 19:47:50.628294 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.628184    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle\") pod \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\" (UID: \"825d9b60-892f-4110-b8b9-ebd26a38f4a3\") "
Oct 30 19:47:50.628843 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.628819    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle" pod "825d9b60-892f-4110-b8b9-ebd26a38f4a3" (UID: "825d9b60-892f-4110-b8b9-ebd26a38f4a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:50.630407 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.630384    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc" pod "825d9b60-892f-4110-b8b9-ebd26a38f4a3" (UID: "825d9b60-892f-4110-b8b9-ebd26a38f4a3"). InnerVolumeSpecName "kube-api-access-tvmqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:50.632702 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.632681    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util" pod "825d9b60-892f-4110-b8b9-ebd26a38f4a3" (UID: "825d9b60-892f-4110-b8b9-ebd26a38f4a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:50.729285 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.729200    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:47:50.729285 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.729231    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmqc\" (UniqueName: \"kubernetes.io/projected/825d9b60-892f-4110-b8b9-ebd26a38f4a3-kube-api-access-tvmqc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:47:50.729285 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:50.729242    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/825d9b60-892f-4110-b8b9-ebd26a38f4a3-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.438025 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:51.437982    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr" event={"ID":"825d9b60-892f-4110-b8b9-ebd26a38f4a3","Type":"ContainerDied","Data":"3a8f02697e59a2bee9308ec3eddca7789511007199d21d65e67e97db1e8139c8"}
Oct 30 19:47:51.438025 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:51.438017    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr6lxr"
Oct 30 19:47:51.438207 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:51.438042    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8f02697e59a2bee9308ec3eddca7789511007199d21d65e67e97db1e8139c8"
Oct 30 19:47:56.843051 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843016    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"]
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843300    4917 cpu_manager.go:414] "containerMap: removing container" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="pull"
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843310    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="pull"
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843323    4917 cpu_manager.go:414] "containerMap: removing container" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="extract"
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843328    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="extract"
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843347    4917 cpu_manager.go:414] "containerMap: removing container" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="util"
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843353    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="util"
Oct 30 19:47:56.843443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.843405    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="825d9b60-892f-4110-b8b9-ebd26a38f4a3" containerName="extract"
Oct 30 19:47:56.865163 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.865140    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:56.899347 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.899318    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"]
Oct 30 19:47:56.948826 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.948791    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6fx7w\""
Oct 30 19:47:56.948977 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.948950    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:47:56.948977 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.948950    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 19:47:56.979979 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.979950    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zw6c\" (UniqueName: \"kubernetes.io/projected/e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6-kube-api-access-5zw6c\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5zp9l\" (UID: \"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:56.980032 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:56.980004    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5zp9l\" (UID: \"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:57.080899 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:57.080864    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5zp9l\" (UID: \"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:57.081056 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:57.080940    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zw6c\" (UniqueName: \"kubernetes.io/projected/e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6-kube-api-access-5zw6c\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5zp9l\" (UID: \"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:57.081260 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:57.081238    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5zp9l\" (UID: \"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:57.148280 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:57.148213    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zw6c\" (UniqueName: \"kubernetes.io/projected/e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6-kube-api-access-5zw6c\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-5zp9l\" (UID: \"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:57.190200 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:57.190170    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"
Oct 30 19:47:57.650510 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:57.650451    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l"]
Oct 30 19:47:57.653741 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:47:57.653712    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1efeb3f_dbc9_4a84_8bcb_1da61edb3ff6.slice/crio-5dde863e67f5375f70f00c897cecfdb3cbf1b084330e22a7cfc95ccf93378ec2 WatchSource:0}: Error finding container 5dde863e67f5375f70f00c897cecfdb3cbf1b084330e22a7cfc95ccf93378ec2: Status 404 returned error can't find the container with id 5dde863e67f5375f70f00c897cecfdb3cbf1b084330e22a7cfc95ccf93378ec2
Oct 30 19:47:58.464380 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:47:58.464339    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l" event={"ID":"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6","Type":"ContainerStarted","Data":"5dde863e67f5375f70f00c897cecfdb3cbf1b084330e22a7cfc95ccf93378ec2"}
Oct 30 19:48:01.477727 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:01.477692    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l" event={"ID":"e1efeb3f-dbc9-4a84-8bcb-1da61edb3ff6","Type":"ContainerStarted","Data":"1d5d7f3fbfd53d5b09c6b96f413ca553e14f073362b001032215466422c85dba"}
Oct 30 19:48:01.757691 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:01.757586    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-5zp9l" podStartSLOduration=2.676319202 podStartE2EDuration="5.757570721s" totalImagesPullingTime="3.081251519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:01.706851759 +0000 UTC m=+377.871253401" watchObservedRunningTime="2026-10-30 19:48:01.757570721 +0000 UTC m=+377.921972360"
Oct 30 19:48:03.441231 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.441195    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"]
Oct 30 19:48:03.444900 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.444865    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.517400 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.517364    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-5fkzj\""
Oct 30 19:48:03.534243 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.534207    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc52r\" (UniqueName: \"kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.534395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.534341    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.534395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.534391    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.561458 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.561424    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:48:03.561458 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.561455    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:48:03.562663 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.562639    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"]
Oct 30 19:48:03.635165 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.635126    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.635334 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.635182    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc52r\" (UniqueName: \"kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.635334 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.635224    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.635602 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.635584    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.635642 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.635585    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.703810 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.703715    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc52r\" (UniqueName: \"kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:03.758595 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:03.758562    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:04.165870 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:04.165680    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"]
Oct 30 19:48:04.490692 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:04.490661    4917 generic.go:356] "Generic (PLEG): container finished" podID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerID="5b42341b41fb90cb07dfdb0dcfce20a37e7f51d0a8597190e47935b7995f4e24" exitCode=0
Oct 30 19:48:04.491104 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:04.490725    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj" event={"ID":"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868","Type":"ContainerDied","Data":"5b42341b41fb90cb07dfdb0dcfce20a37e7f51d0a8597190e47935b7995f4e24"}
Oct 30 19:48:04.491104 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:04.490760    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj" event={"ID":"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868","Type":"ContainerStarted","Data":"4ebf8fbd83236797efc478563ca77c863b5d1bade1b240e522176ede1921966f"}
Oct 30 19:48:07.504387 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:07.504350    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj" event={"ID":"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868","Type":"ContainerStarted","Data":"997b2c0a50d373a8552b6680ca7d0fa69b603f55e9c3e42b41b463e4fb588433"}
Oct 30 19:48:08.509974 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:08.509937    4917 generic.go:356] "Generic (PLEG): container finished" podID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerID="997b2c0a50d373a8552b6680ca7d0fa69b603f55e9c3e42b41b463e4fb588433" exitCode=0
Oct 30 19:48:08.510335 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:08.510019    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj" event={"ID":"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868","Type":"ContainerDied","Data":"997b2c0a50d373a8552b6680ca7d0fa69b603f55e9c3e42b41b463e4fb588433"}
Oct 30 19:48:09.516589 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:09.516557    4917 generic.go:356] "Generic (PLEG): container finished" podID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerID="0cdc6d7aaa97836d216d962eaed6a77ca3d0f8985389775398331af4e4273c5c" exitCode=0
Oct 30 19:48:09.517005 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:09.516618    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj" event={"ID":"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868","Type":"ContainerDied","Data":"0cdc6d7aaa97836d216d962eaed6a77ca3d0f8985389775398331af4e4273c5c"}
Oct 30 19:48:09.679450 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:48:09.679406    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:48:10.642340 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.642315    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:10.797496 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.797412    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle\") pod \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") "
Oct 30 19:48:10.797655 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.797500    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r\" (UniqueName: \"kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r\") pod \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") "
Oct 30 19:48:10.797655 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.797531    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util\") pod \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\" (UID: \"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868\") "
Oct 30 19:48:10.797959 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.797917    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle" pod "bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" (UID: "bd61e2ca-b9cc-4fb8-a8d8-8c526c812868"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:10.799686 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.799667    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r" pod "bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" (UID: "bd61e2ca-b9cc-4fb8-a8d8-8c526c812868"). InnerVolumeSpecName "kube-api-access-dc52r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:10.801888 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.801862    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util" pod "bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" (UID: "bd61e2ca-b9cc-4fb8-a8d8-8c526c812868"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:10.898492 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.898458    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc52r\" (UniqueName: \"kubernetes.io/projected/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-kube-api-access-dc52r\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:10.898492 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.898485    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:10.898492 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:10.898495    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd61e2ca-b9cc-4fb8-a8d8-8c526c812868-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.525444 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:11.525416    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj"
Oct 30 19:48:11.525444 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:11.525424    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85zwcjj" event={"ID":"bd61e2ca-b9cc-4fb8-a8d8-8c526c812868","Type":"ContainerDied","Data":"4ebf8fbd83236797efc478563ca77c863b5d1bade1b240e522176ede1921966f"}
Oct 30 19:48:11.525649 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:11.525471    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ebf8fbd83236797efc478563ca77c863b5d1bade1b240e522176ede1921966f"
Oct 30 19:48:13.604580 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604544    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"]
Oct 30 19:48:13.604995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604900    4917 cpu_manager.go:414] "containerMap: removing container" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="util"
Oct 30 19:48:13.604995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604911    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="util"
Oct 30 19:48:13.604995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604922    4917 cpu_manager.go:414] "containerMap: removing container" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="pull"
Oct 30 19:48:13.604995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604927    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="pull"
Oct 30 19:48:13.604995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604934    4917 cpu_manager.go:414] "containerMap: removing container" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="extract"
Oct 30 19:48:13.604995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.604940    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="extract"
Oct 30 19:48:13.605181 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.605009    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd61e2ca-b9cc-4fb8-a8d8-8c526c812868" containerName="extract"
Oct 30 19:48:13.606881 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.606865    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.660904 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.660872    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hh9hj\""
Oct 30 19:48:13.661057 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.661038    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 19:48:13.711536 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.711508    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 19:48:13.712940 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.712916    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"]
Oct 30 19:48:13.722057 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.722034    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7sk\" (UniqueName: \"kubernetes.io/projected/16655b4a-806f-43b4-812a-9782b3237211-kube-api-access-fq7sk\") pod \"cert-manager-webhook-5dbbbf75cc-2vvxc\" (UID: \"16655b4a-806f-43b4-812a-9782b3237211\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.722139 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.722071    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16655b4a-806f-43b4-812a-9782b3237211-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2vvxc\" (UID: \"16655b4a-806f-43b4-812a-9782b3237211\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.823055 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.823018    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7sk\" (UniqueName: \"kubernetes.io/projected/16655b4a-806f-43b4-812a-9782b3237211-kube-api-access-fq7sk\") pod \"cert-manager-webhook-5dbbbf75cc-2vvxc\" (UID: \"16655b4a-806f-43b4-812a-9782b3237211\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.823250 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.823094    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16655b4a-806f-43b4-812a-9782b3237211-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2vvxc\" (UID: \"16655b4a-806f-43b4-812a-9782b3237211\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.887645 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.887564    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16655b4a-806f-43b4-812a-9782b3237211-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2vvxc\" (UID: \"16655b4a-806f-43b4-812a-9782b3237211\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.887799 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.887760    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7sk\" (UniqueName: \"kubernetes.io/projected/16655b4a-806f-43b4-812a-9782b3237211-kube-api-access-fq7sk\") pod \"cert-manager-webhook-5dbbbf75cc-2vvxc\" (UID: \"16655b4a-806f-43b4-812a-9782b3237211\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:13.931974 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:13.931932    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:14.318710 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:14.318610    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"]
Oct 30 19:48:14.321366 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:48:14.321341    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16655b4a_806f_43b4_812a_9782b3237211.slice/crio-c446fefd21087ff73dfdab25ac3fc691fce09d981cde083e42f8387365382701 WatchSource:0}: Error finding container c446fefd21087ff73dfdab25ac3fc691fce09d981cde083e42f8387365382701: Status 404 returned error can't find the container with id c446fefd21087ff73dfdab25ac3fc691fce09d981cde083e42f8387365382701
Oct 30 19:48:14.539842 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:14.539799    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc" event={"ID":"16655b4a-806f-43b4-812a-9782b3237211","Type":"ContainerStarted","Data":"c446fefd21087ff73dfdab25ac3fc691fce09d981cde083e42f8387365382701"}
Oct 30 19:48:21.569245 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:21.569196    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc" event={"ID":"16655b4a-806f-43b4-812a-9782b3237211","Type":"ContainerStarted","Data":"92f34c2febe944388b345847f7f26066185597d033dced35f9f71c28fdd37a6a"}
Oct 30 19:48:21.569684 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:21.569340    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:21.689180 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:21.689124    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc" podStartSLOduration=1.9628738719999999 podStartE2EDuration="8.689109655s" totalImagesPullingTime="6.726235783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:21.687039582 +0000 UTC m=+397.851441223" watchObservedRunningTime="2026-10-30 19:48:21.689109655 +0000 UTC m=+397.853511296"
Oct 30 19:48:23.897407 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:23.897333    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"]
Oct 30 19:48:23.899650 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:23.899634    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:23.994250 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:23.994216    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:23.994250 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:23.994253    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:23.994434 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:23.994339    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbcb\" (UniqueName: \"kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.003474 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.003448    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ggc82\""
Oct 30 19:48:24.003605 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.003513    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:48:24.003605 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.003520    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:48:24.004956 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.004938    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"]
Oct 30 19:48:24.095496 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.095459    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbcb\" (UniqueName: \"kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.095660 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.095516    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.095660 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.095547    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.095942 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.095925    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.096010 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.095973    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.158322 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.158252    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbcb\" (UniqueName: \"kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.211791 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.211744    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:24.587025 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:24.586990    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"]
Oct 30 19:48:24.590717 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:48:24.590690    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb926ab13_882a_40d1_a0ed_5a51b8cb7375.slice/crio-7621afbb40a9f7ca17fccfd96494ef80bcdc01cf876885c8065e7dd4f535ed79 WatchSource:0}: Error finding container 7621afbb40a9f7ca17fccfd96494ef80bcdc01cf876885c8065e7dd4f535ed79: Status 404 returned error can't find the container with id 7621afbb40a9f7ca17fccfd96494ef80bcdc01cf876885c8065e7dd4f535ed79
Oct 30 19:48:25.587949 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:25.587916    4917 generic.go:356] "Generic (PLEG): container finished" podID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerID="bb2a40c3818b1185d03b5870e7b914f8fe98f854852f18e9606b47109729e769" exitCode=0
Oct 30 19:48:25.588393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:25.588010    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk" event={"ID":"b926ab13-882a-40d1-a0ed-5a51b8cb7375","Type":"ContainerDied","Data":"bb2a40c3818b1185d03b5870e7b914f8fe98f854852f18e9606b47109729e769"}
Oct 30 19:48:25.588393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:25.588072    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk" event={"ID":"b926ab13-882a-40d1-a0ed-5a51b8cb7375","Type":"ContainerStarted","Data":"7621afbb40a9f7ca17fccfd96494ef80bcdc01cf876885c8065e7dd4f535ed79"}
Oct 30 19:48:27.576206 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:27.576176    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc"
Oct 30 19:48:27.597928 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:27.597893    4917 generic.go:356] "Generic (PLEG): container finished" podID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerID="c40f40b4ae64da23e38eb193ca5ee9f131eea685b3b232a5475abc7e93b6e277" exitCode=0
Oct 30 19:48:27.598071 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:27.597974    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk" event={"ID":"b926ab13-882a-40d1-a0ed-5a51b8cb7375","Type":"ContainerDied","Data":"c40f40b4ae64da23e38eb193ca5ee9f131eea685b3b232a5475abc7e93b6e277"}
Oct 30 19:48:28.604007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:28.603921    4917 generic.go:356] "Generic (PLEG): container finished" podID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerID="bc6c9cbc4ee956b0e39f20d75a13bdf9289c87d0fef84a2085298ae61aa681ee" exitCode=0
Oct 30 19:48:28.604405 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:28.604006    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk" event={"ID":"b926ab13-882a-40d1-a0ed-5a51b8cb7375","Type":"ContainerDied","Data":"bc6c9cbc4ee956b0e39f20d75a13bdf9289c87d0fef84a2085298ae61aa681ee"}
Oct 30 19:48:29.736656 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.736632    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:29.838221 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.838187    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util\") pod \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") "
Oct 30 19:48:29.838412 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.838297    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb\" (UniqueName: \"kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb\") pod \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") "
Oct 30 19:48:29.838412 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.838340    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle\") pod \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\" (UID: \"b926ab13-882a-40d1-a0ed-5a51b8cb7375\") "
Oct 30 19:48:29.838903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.838872    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle" pod "b926ab13-882a-40d1-a0ed-5a51b8cb7375" (UID: "b926ab13-882a-40d1-a0ed-5a51b8cb7375"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.840398 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.840379    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb" pod "b926ab13-882a-40d1-a0ed-5a51b8cb7375" (UID: "b926ab13-882a-40d1-a0ed-5a51b8cb7375"). InnerVolumeSpecName "kube-api-access-gfbcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:29.843104 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.843080    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util" pod "b926ab13-882a-40d1-a0ed-5a51b8cb7375" (UID: "b926ab13-882a-40d1-a0ed-5a51b8cb7375"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.939832 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.939801    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.939832 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.939828    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b926ab13-882a-40d1-a0ed-5a51b8cb7375-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.939995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:29.939842    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbcb\" (UniqueName: \"kubernetes.io/projected/b926ab13-882a-40d1-a0ed-5a51b8cb7375-kube-api-access-gfbcb\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:30.613519 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:30.613484    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk" event={"ID":"b926ab13-882a-40d1-a0ed-5a51b8cb7375","Type":"ContainerDied","Data":"7621afbb40a9f7ca17fccfd96494ef80bcdc01cf876885c8065e7dd4f535ed79"}
Oct 30 19:48:30.613691 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:30.613530    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7621afbb40a9f7ca17fccfd96494ef80bcdc01cf876885c8065e7dd4f535ed79"
Oct 30 19:48:30.613691 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:30.613530    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7fpk"
Oct 30 19:48:39.460327 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460288    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"]
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460787    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="util"
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460806    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="util"
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460820    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="pull"
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460828    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="pull"
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460840    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="extract"
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460847    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="extract"
Oct 30 19:48:39.461001 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.460937    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b926ab13-882a-40d1-a0ed-5a51b8cb7375" containerName="extract"
Oct 30 19:48:39.467725 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.467703    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.512949 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.512917    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdpm\" (UniqueName: \"kubernetes.io/projected/29cef479-2bf6-49a6-b680-4da0a7c44b56-kube-api-access-9zdpm\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.513098 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.512964    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.513098 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.513041    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.513183 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.513126    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29cef479-2bf6-49a6-b680-4da0a7c44b56-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.513183 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.513153    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cef479-2bf6-49a6-b680-4da0a7c44b56-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.513183 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.513169    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.525042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.525014    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:48:39.525042 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.525014    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:48:39.525193 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.525078    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:48:39.571777 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.571739    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:48:39.571777 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.571766    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vn6w\""
Oct 30 19:48:39.571949 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.571871    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:48:39.605930 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.605902    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"]
Oct 30 19:48:39.614241 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614217    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdpm\" (UniqueName: \"kubernetes.io/projected/29cef479-2bf6-49a6-b680-4da0a7c44b56-kube-api-access-9zdpm\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.614363 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614253    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.614363 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614282    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.614363 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614311    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29cef479-2bf6-49a6-b680-4da0a7c44b56-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.614363 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614336    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cef479-2bf6-49a6-b680-4da0a7c44b56-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.614572 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614362    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.614694 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.614668    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29cef479-2bf6-49a6-b680-4da0a7c44b56-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.616691 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.616662    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cef479-2bf6-49a6-b680-4da0a7c44b56-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.616938 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.616920    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.617248 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.617223    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.617443 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.617427    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29cef479-2bf6-49a6-b680-4da0a7c44b56-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.678818 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:48:39.678766    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:48:39.692256 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.692226    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdpm\" (UniqueName: \"kubernetes.io/projected/29cef479-2bf6-49a6-b680-4da0a7c44b56-kube-api-access-9zdpm\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps\" (UID: \"29cef479-2bf6-49a6-b680-4da0a7c44b56\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:39.782917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:39.782835    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:40.314662 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:40.314631    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"]
Oct 30 19:48:40.317835 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:48:40.317809    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cef479_2bf6_49a6_b680_4da0a7c44b56.slice/crio-ae39aa486562d4188a2d6fb86da93d8ff88ae4a6da36dd230166a72fde77cf55 WatchSource:0}: Error finding container ae39aa486562d4188a2d6fb86da93d8ff88ae4a6da36dd230166a72fde77cf55: Status 404 returned error can't find the container with id ae39aa486562d4188a2d6fb86da93d8ff88ae4a6da36dd230166a72fde77cf55
Oct 30 19:48:40.655131 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:40.655055    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps" event={"ID":"29cef479-2bf6-49a6-b680-4da0a7c44b56","Type":"ContainerStarted","Data":"ae39aa486562d4188a2d6fb86da93d8ff88ae4a6da36dd230166a72fde77cf55"}
Oct 30 19:48:44.673571 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:44.673536    4917 generic.go:356] "Generic (PLEG): container finished" podID="29cef479-2bf6-49a6-b680-4da0a7c44b56" containerID="f83e53c43cb2c218548f5db62e4de9d9a2f87b5318b1791a2f2092cb08c2b70c" exitCode=0
Oct 30 19:48:44.673964 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:44.673619    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps" event={"ID":"29cef479-2bf6-49a6-b680-4da0a7c44b56","Type":"ContainerDied","Data":"f83e53c43cb2c218548f5db62e4de9d9a2f87b5318b1791a2f2092cb08c2b70c"}
Oct 30 19:48:46.682542 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:46.682506    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps" event={"ID":"29cef479-2bf6-49a6-b680-4da0a7c44b56","Type":"ContainerStarted","Data":"d13f58c1db809f88a292515e268696b3034bab909bd7d2c50947f037da563521"}
Oct 30 19:48:46.683000 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:46.682751    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:48:46.936236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:46.936134    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps" podStartSLOduration=2.89528259 podStartE2EDuration="7.936120732s" totalImagesPullingTime="4.571135951s" totalInitContainerRuntime="469.702191ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:46.885098008 +0000 UTC m=+423.049499671" watchObservedRunningTime="2026-10-30 19:48:46.936120732 +0000 UTC m=+423.100522374"
Oct 30 19:48:57.691785 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:48:57.691737    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps"
Oct 30 19:49:00.677522 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.677486    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"]
Oct 30 19:49:00.680684 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.680667    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.782989 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.782955    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a198a1d9-9722-4857-9890-383e8f756e55-metrics-cert\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.783155 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.783000    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a198a1d9-9722-4857-9890-383e8f756e55-manager-config\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.783155 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.783026    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hrn\" (UniqueName: \"kubernetes.io/projected/a198a1d9-9722-4857-9890-383e8f756e55-kube-api-access-b7hrn\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.783155 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.783116    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a198a1d9-9722-4857-9890-383e8f756e55-cert\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.789518 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.789499    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 19:49:00.789575 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.789538    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 19:49:00.789575 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.789569    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mzftv\""
Oct 30 19:49:00.789683 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.789671    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:49:00.790086 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.790075    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 19:49:00.790139 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.790078    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:49:00.791018 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.791002    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"]
Oct 30 19:49:00.883521 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.883488    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a198a1d9-9722-4857-9890-383e8f756e55-manager-config\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.883521 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.883524    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hrn\" (UniqueName: \"kubernetes.io/projected/a198a1d9-9722-4857-9890-383e8f756e55-kube-api-access-b7hrn\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.883819 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.883558    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a198a1d9-9722-4857-9890-383e8f756e55-cert\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.883819 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.883615    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a198a1d9-9722-4857-9890-383e8f756e55-metrics-cert\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.884308 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.884251    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a198a1d9-9722-4857-9890-383e8f756e55-manager-config\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.886358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.886328    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a198a1d9-9722-4857-9890-383e8f756e55-cert\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.886461 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.886360    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a198a1d9-9722-4857-9890-383e8f756e55-metrics-cert\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.946432 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.946358    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hrn\" (UniqueName: \"kubernetes.io/projected/a198a1d9-9722-4857-9890-383e8f756e55-kube-api-access-b7hrn\") pod \"lws-controller-manager-5794f57dfd-c7jn8\" (UID: \"a198a1d9-9722-4857-9890-383e8f756e55\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:00.992145 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:00.992114    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:01.369398 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:01.369272    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"]
Oct 30 19:49:01.372282 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:49:01.372246    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda198a1d9_9722_4857_9890_383e8f756e55.slice/crio-474b5b7ee3bfccd865276cb4d7989fb10bccd7a0df7f24291579f5dbffc7f684 WatchSource:0}: Error finding container 474b5b7ee3bfccd865276cb4d7989fb10bccd7a0df7f24291579f5dbffc7f684: Status 404 returned error can't find the container with id 474b5b7ee3bfccd865276cb4d7989fb10bccd7a0df7f24291579f5dbffc7f684
Oct 30 19:49:01.750973 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:01.750935    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8" event={"ID":"a198a1d9-9722-4857-9890-383e8f756e55","Type":"ContainerStarted","Data":"474b5b7ee3bfccd865276cb4d7989fb10bccd7a0df7f24291579f5dbffc7f684"}
Oct 30 19:49:04.765965 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:04.765870    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8" event={"ID":"a198a1d9-9722-4857-9890-383e8f756e55","Type":"ContainerStarted","Data":"f377e089a2703b14acad7646e899fbf2a19f057b602f36d64f325e284c37e111"}
Oct 30 19:49:04.766353 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:04.766118    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:04.898393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:04.898332    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8" podStartSLOduration=1.761159371 podStartE2EDuration="4.898318264s" totalImagesPullingTime="3.137158893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:04.897150389 +0000 UTC m=+441.061552033" watchObservedRunningTime="2026-10-30 19:49:04.898318264 +0000 UTC m=+441.062719906"
Oct 30 19:49:09.678128 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:49:09.678091    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:49:12.288685 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.288651    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"]
Oct 30 19:49:12.292108 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.292092    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.345707 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.345679    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:49:12.367536 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.367510    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6k28l\""
Oct 30 19:49:12.386043 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386006    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386225 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386073    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386225 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386141    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386225 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386192    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386361 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386260    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386361 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386296    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386457 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386376    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386457 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386422    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.386553 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.386455    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fgj\" (UniqueName: \"kubernetes.io/projected/a75eeff5-1531-4c77-9c57-b2bcd81d5871-kube-api-access-q7fgj\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.399393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.399357    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"]
Oct 30 19:49:12.490085 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490034    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490292 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490101    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490292 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490141    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490292 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490189    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490292 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490232    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490292 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490266    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490520 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490295    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490520 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490325    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fgj\" (UniqueName: \"kubernetes.io/projected/a75eeff5-1531-4c77-9c57-b2bcd81d5871-kube-api-access-q7fgj\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490520 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490429    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.490947 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490915    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.491087 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.490953    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.491317 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.491206    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.491317 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.491262    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.491317 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.491276    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.492951 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.492926    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.493149 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.493130    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.554630 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.554553    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fgj\" (UniqueName: \"kubernetes.io/projected/a75eeff5-1531-4c77-9c57-b2bcd81d5871-kube-api-access-q7fgj\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.558418 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.558392    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75eeff5-1531-4c77-9c57-b2bcd81d5871-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-h9wmt\" (UID: \"a75eeff5-1531-4c77-9c57-b2bcd81d5871\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:12.603756 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:12.603720    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:13.028758 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:13.028727    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"]
Oct 30 19:49:13.031922 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:49:13.031892    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75eeff5_1531_4c77_9c57_b2bcd81d5871.slice/crio-ce04019b1dddfa98a0a823c5aefc1345eae95448b69d763bb708164473f384a0 WatchSource:0}: Error finding container ce04019b1dddfa98a0a823c5aefc1345eae95448b69d763bb708164473f384a0: Status 404 returned error can't find the container with id ce04019b1dddfa98a0a823c5aefc1345eae95448b69d763bb708164473f384a0
Oct 30 19:49:13.802633 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:13.802598    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt" event={"ID":"a75eeff5-1531-4c77-9c57-b2bcd81d5871","Type":"ContainerStarted","Data":"ce04019b1dddfa98a0a823c5aefc1345eae95448b69d763bb708164473f384a0"}
Oct 30 19:49:15.772989 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:15.772960    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-c7jn8"
Oct 30 19:49:17.014209 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:17.014183    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:17.014446 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:17.014249    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:17.014446 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:17.014275    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:49:17.824128 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:17.824091    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt" event={"ID":"a75eeff5-1531-4c77-9c57-b2bcd81d5871","Type":"ContainerStarted","Data":"87cf037ef00bed2ab9654b790ebd6d699c58863680e679ad81b460e8779233c8"}
Oct 30 19:49:17.997989 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:17.997933    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt" podStartSLOduration=2.018332199 podStartE2EDuration="5.997920636s" totalImagesPullingTime="3.979588437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:17.996697639 +0000 UTC m=+454.161099274" watchObservedRunningTime="2026-10-30 19:49:17.997920636 +0000 UTC m=+454.162322279"
Oct 30 19:49:18.604877 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:18.604834    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:18.609431 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:18.609403    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:18.828205 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:18.828167    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:18.829130 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:18.829112    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-h9wmt"
Oct 30 19:49:32.368127 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.368092    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"]
Oct 30 19:49:32.371454 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.371437    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.446945 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.446904    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/625ee131-9ed8-4e2e-a591-66298aed7e86-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.447092 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.447025    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5jn\" (UniqueName: \"kubernetes.io/projected/625ee131-9ed8-4e2e-a591-66298aed7e86-kube-api-access-kw5jn\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.447173 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.447120    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/625ee131-9ed8-4e2e-a591-66298aed7e86-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.447411 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.447383    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/625ee131-9ed8-4e2e-a591-66298aed7e86-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.475615 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.475584    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gdq9r\""
Oct 30 19:49:32.548461 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.548428    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/625ee131-9ed8-4e2e-a591-66298aed7e86-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.548461 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.548461    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5jn\" (UniqueName: \"kubernetes.io/projected/625ee131-9ed8-4e2e-a591-66298aed7e86-kube-api-access-kw5jn\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.548651 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.548486    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/625ee131-9ed8-4e2e-a591-66298aed7e86-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.548651 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.548516    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/625ee131-9ed8-4e2e-a591-66298aed7e86-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.548855 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.548834    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/625ee131-9ed8-4e2e-a591-66298aed7e86-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.548895 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.548867    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/625ee131-9ed8-4e2e-a591-66298aed7e86-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.577251 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.577231    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 19:49:32.578937 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.578918    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"]
Oct 30 19:49:32.579557 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.579538    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/625ee131-9ed8-4e2e-a591-66298aed7e86-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.614242 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.614219    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5jn\" (UniqueName: \"kubernetes.io/projected/625ee131-9ed8-4e2e-a591-66298aed7e86-kube-api-access-kw5jn\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rp7h\" (UID: \"625ee131-9ed8-4e2e-a591-66298aed7e86\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:32.683859 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:32.683836    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:33.117299 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:33.117268    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"]
Oct 30 19:49:33.119715 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:49:33.119687    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625ee131_9ed8_4e2e_a591_66298aed7e86.slice/crio-e240053a0b8c4e35a34c75c5bb2c4f0b874e621b71495ae75025fb43eaeed4f2 WatchSource:0}: Error finding container e240053a0b8c4e35a34c75c5bb2c4f0b874e621b71495ae75025fb43eaeed4f2: Status 404 returned error can't find the container with id e240053a0b8c4e35a34c75c5bb2c4f0b874e621b71495ae75025fb43eaeed4f2
Oct 30 19:49:33.891743 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:33.891704    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h" event={"ID":"625ee131-9ed8-4e2e-a591-66298aed7e86","Type":"ContainerStarted","Data":"e240053a0b8c4e35a34c75c5bb2c4f0b874e621b71495ae75025fb43eaeed4f2"}
Oct 30 19:49:36.907331 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:36.907293    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h" event={"ID":"625ee131-9ed8-4e2e-a591-66298aed7e86","Type":"ContainerStarted","Data":"f09faf870d84e9f4132c2c5dd6d9b738ecc01a631c79b003007565e6b4012ef1"}
Oct 30 19:49:36.907722 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:36.907413    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:49:37.088135 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:37.088082    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h" podStartSLOduration=2.3559535670000002 podStartE2EDuration="5.08806843s" totalImagesPullingTime="2.732114863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:37.078643022 +0000 UTC m=+473.243044663" watchObservedRunningTime="2026-10-30 19:49:37.08806843 +0000 UTC m=+473.252470094"
Oct 30 19:49:39.678194 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:49:39.678157    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:49:57.915008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:49:57.914937    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rp7h"
Oct 30 19:50:09.678594 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:50:09.678556    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:50:32.132806 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.132755    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xht5c"]
Oct 30 19:50:32.136382 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.136365    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.144896 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.144871    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mq8\" (UniqueName: \"kubernetes.io/projected/52325904-1a6d-4d6b-9766-132c039f063f-kube-api-access-w7mq8\") pod \"kserve-controller-manager-776cc66d47-xht5c\" (UID: \"52325904-1a6d-4d6b-9766-132c039f063f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.144992 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.144975    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52325904-1a6d-4d6b-9766-132c039f063f-cert\") pod \"kserve-controller-manager-776cc66d47-xht5c\" (UID: \"52325904-1a6d-4d6b-9766-132c039f063f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.189620 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.189591    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7b8xs\""
Oct 30 19:50:32.194738 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.194708    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 19:50:32.196894 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.196871    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xht5c"]
Oct 30 19:50:32.245560 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.245523    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52325904-1a6d-4d6b-9766-132c039f063f-cert\") pod \"kserve-controller-manager-776cc66d47-xht5c\" (UID: \"52325904-1a6d-4d6b-9766-132c039f063f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.245716 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.245583    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mq8\" (UniqueName: \"kubernetes.io/projected/52325904-1a6d-4d6b-9766-132c039f063f-kube-api-access-w7mq8\") pod \"kserve-controller-manager-776cc66d47-xht5c\" (UID: \"52325904-1a6d-4d6b-9766-132c039f063f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.248156 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.248134    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52325904-1a6d-4d6b-9766-132c039f063f-cert\") pod \"kserve-controller-manager-776cc66d47-xht5c\" (UID: \"52325904-1a6d-4d6b-9766-132c039f063f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.255237 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.255215    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"]
Oct 30 19:50:32.258576 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.258558    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.313418 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.313384    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:50:32.313550 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.313485    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lqjjd\""
Oct 30 19:50:32.317140 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.317118    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mq8\" (UniqueName: \"kubernetes.io/projected/52325904-1a6d-4d6b-9766-132c039f063f-kube-api-access-w7mq8\") pod \"kserve-controller-manager-776cc66d47-xht5c\" (UID: \"52325904-1a6d-4d6b-9766-132c039f063f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.321544 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.321508    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"]
Oct 30 19:50:32.346915 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.346871    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6h7\" (UniqueName: \"kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7\") pod \"llmisvc-controller-manager-869d5dcc5d-8kc6z\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.347066 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.346962    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8kc6z\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.378273 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.378246    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wwt22"]
Oct 30 19:50:32.381570 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.381555    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.435014 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.434991    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 19:50:32.435014 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.434988    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8ckw5\""
Oct 30 19:50:32.442523 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.442495    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wwt22"]
Oct 30 19:50:32.447454 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.447430    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6h7\" (UniqueName: \"kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7\") pod \"llmisvc-controller-manager-869d5dcc5d-8kc6z\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.447567 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.447462    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8kc6z\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.447567 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.447488    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c297f662-710f-4a22-af65-ca401969d921-tls-certs\") pod \"model-serving-api-75bd89fdfc-wwt22\" (UID: \"c297f662-710f-4a22-af65-ca401969d921\") " pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.447567 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.447522    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvxv\" (UniqueName: \"kubernetes.io/projected/c297f662-710f-4a22-af65-ca401969d921-kube-api-access-6zvxv\") pod \"model-serving-api-75bd89fdfc-wwt22\" (UID: \"c297f662-710f-4a22-af65-ca401969d921\") " pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.448469 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.448446    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:32.449943 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.449923    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8kc6z\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.528159 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.528128    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6h7\" (UniqueName: \"kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7\") pod \"llmisvc-controller-manager-869d5dcc5d-8kc6z\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.548191 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.548141    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c297f662-710f-4a22-af65-ca401969d921-tls-certs\") pod \"model-serving-api-75bd89fdfc-wwt22\" (UID: \"c297f662-710f-4a22-af65-ca401969d921\") " pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.548575 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.548236    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvxv\" (UniqueName: \"kubernetes.io/projected/c297f662-710f-4a22-af65-ca401969d921-kube-api-access-6zvxv\") pod \"model-serving-api-75bd89fdfc-wwt22\" (UID: \"c297f662-710f-4a22-af65-ca401969d921\") " pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.550477 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.550455    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c297f662-710f-4a22-af65-ca401969d921-tls-certs\") pod \"model-serving-api-75bd89fdfc-wwt22\" (UID: \"c297f662-710f-4a22-af65-ca401969d921\") " pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.570297 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.570274    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:32.620312 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.620281    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvxv\" (UniqueName: \"kubernetes.io/projected/c297f662-710f-4a22-af65-ca401969d921-kube-api-access-6zvxv\") pod \"model-serving-api-75bd89fdfc-wwt22\" (UID: \"c297f662-710f-4a22-af65-ca401969d921\") " pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.694064 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.694033    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:32.827091 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.827066    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xht5c"]
Oct 30 19:50:32.829263 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:50:32.829218    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52325904_1a6d_4d6b_9766_132c039f063f.slice/crio-7a854ca07b42f12902d379481e704cd52136a23c0a0085266f96f8f9133d44d5 WatchSource:0}: Error finding container 7a854ca07b42f12902d379481e704cd52136a23c0a0085266f96f8f9133d44d5: Status 404 returned error can't find the container with id 7a854ca07b42f12902d379481e704cd52136a23c0a0085266f96f8f9133d44d5
Oct 30 19:50:32.940425 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:32.940391    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"]
Oct 30 19:50:32.943079 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:50:32.943050    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80317fab_980b_49a3_97eb_1ea5a577d79d.slice/crio-254d7d6efe92dee63c6a26bab6112dfe99ae6af8d17a5741bb0053bd62fcaad5 WatchSource:0}: Error finding container 254d7d6efe92dee63c6a26bab6112dfe99ae6af8d17a5741bb0053bd62fcaad5: Status 404 returned error can't find the container with id 254d7d6efe92dee63c6a26bab6112dfe99ae6af8d17a5741bb0053bd62fcaad5
Oct 30 19:50:33.072068 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:33.072043    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wwt22"]
Oct 30 19:50:33.074472 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:50:33.074446    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc297f662_710f_4a22_af65_ca401969d921.slice/crio-ad39bd9e94b0f0a9de023febf2acc7fb4a7d6c28d2e5528bd289eb66b7fa567d WatchSource:0}: Error finding container ad39bd9e94b0f0a9de023febf2acc7fb4a7d6c28d2e5528bd289eb66b7fa567d: Status 404 returned error can't find the container with id ad39bd9e94b0f0a9de023febf2acc7fb4a7d6c28d2e5528bd289eb66b7fa567d
Oct 30 19:50:33.152212 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:33.152168    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c" event={"ID":"52325904-1a6d-4d6b-9766-132c039f063f","Type":"ContainerStarted","Data":"7a854ca07b42f12902d379481e704cd52136a23c0a0085266f96f8f9133d44d5"}
Oct 30 19:50:33.153364 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:33.153337    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22" event={"ID":"c297f662-710f-4a22-af65-ca401969d921","Type":"ContainerStarted","Data":"ad39bd9e94b0f0a9de023febf2acc7fb4a7d6c28d2e5528bd289eb66b7fa567d"}
Oct 30 19:50:33.154459 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:33.154439    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z" event={"ID":"80317fab-980b-49a3-97eb-1ea5a577d79d","Type":"ContainerStarted","Data":"254d7d6efe92dee63c6a26bab6112dfe99ae6af8d17a5741bb0053bd62fcaad5"}
Oct 30 19:50:36.175659 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.175626    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c" event={"ID":"52325904-1a6d-4d6b-9766-132c039f063f","Type":"ContainerStarted","Data":"12f073dc97b85be6b3754d265c66d8eab40f8eee109a22b8c969a1eda67747e1"}
Oct 30 19:50:36.176127 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.175802    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:50:36.180270 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.180238    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22" event={"ID":"c297f662-710f-4a22-af65-ca401969d921","Type":"ContainerStarted","Data":"afa225aaeaf041655bf00e52e50b6d5cd2544cb0d449196780957be3e161eecb"}
Oct 30 19:50:36.180426 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.180297    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:50:36.182070 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.182046    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z" event={"ID":"80317fab-980b-49a3-97eb-1ea5a577d79d","Type":"ContainerStarted","Data":"b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8"}
Oct 30 19:50:36.182177 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.182126    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:50:36.294793 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.294724    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c" podStartSLOduration=1.443080464 podStartE2EDuration="4.294710219s" totalImagesPullingTime="2.851629755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:36.293874265 +0000 UTC m=+532.458275906" watchObservedRunningTime="2026-10-30 19:50:36.294710219 +0000 UTC m=+532.459111860"
Oct 30 19:50:36.412311 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.412260    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z" podStartSLOduration=1.676699861 podStartE2EDuration="4.412247377s" totalImagesPullingTime="2.735547516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:36.411511955 +0000 UTC m=+532.575913597" watchObservedRunningTime="2026-10-30 19:50:36.412247377 +0000 UTC m=+532.576649061"
Oct 30 19:50:36.529688 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:36.529638    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22" podStartSLOduration=1.928257828 podStartE2EDuration="4.529624857s" totalImagesPullingTime="2.601367029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:36.529468948 +0000 UTC m=+532.693870592" watchObservedRunningTime="2026-10-30 19:50:36.529624857 +0000 UTC m=+532.694026489"
Oct 30 19:50:39.677854 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:50:39.677816    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:50:47.190786 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:50:47.190748    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-wwt22"
Oct 30 19:51:07.189419 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:07.189387    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:51:07.192551 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:07.192523    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xht5c"
Oct 30 19:51:09.678231 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:51:09.678195    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:51:39.190257 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.190177    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"]
Oct 30 19:51:39.195168 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.195146    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.299112 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.299084    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.299112 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.299113    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.299319 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.299200    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8fz\" (UniqueName: \"kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.301556 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.301537    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fdxbb\""
Oct 30 19:51:39.301675 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.301591    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:51:39.301675 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.301594    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:51:39.303245 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.303221    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"]
Oct 30 19:51:39.395574 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.395541    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"]
Oct 30 19:51:39.399055 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.399041    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.399565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.399548    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8fz\" (UniqueName: \"kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.399660 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.399604    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.399660 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.399623    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.399978 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.399958    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.400043 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.399988    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.452865 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.452802    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kxdv7\""
Oct 30 19:51:39.459188 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.459161    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"]
Oct 30 19:51:39.465312 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.465286    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8fz\" (UniqueName: \"kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.500228 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.500194    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.500373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.500273    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmk5f\" (UniqueName: \"kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.500373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.500341    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.511155 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.511134    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:39.542817 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.542789    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"]
Oct 30 19:51:39.546429 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.546406    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.599463 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.599428    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rgqqm\""
Oct 30 19:51:39.601824 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.601798    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mn74\" (UniqueName: \"kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.601926 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.601840    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmk5f\" (UniqueName: \"kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.601926 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.601864    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.602021 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.601924    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.602021 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.601964    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.602021 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.601997    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.602225 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.602206    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.602266 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.602229    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.609320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.609294    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"]
Oct 30 19:51:39.655358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.655334    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"]
Oct 30 19:51:39.659017 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.658998    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.675382 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.675356    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmk5f\" (UniqueName: \"kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.679034 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:51:39.679004    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:51:39.702815 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.702793    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mn74\" (UniqueName: \"kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.702964 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.702828    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.702964 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.702851    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jj9\" (UniqueName: \"kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.702964 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.702888    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.702964 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.702944    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.703173 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.702967    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.703247 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.703228    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.703330 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.703313    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.712867 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.712850    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-v82x4\""
Oct 30 19:51:39.715008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.714993    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:39.718648 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.718631    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"]
Oct 30 19:51:39.765917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.765860    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mn74\" (UniqueName: \"kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.804033 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.804003    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.804195 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.804061    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.804195 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.804096    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jj9\" (UniqueName: \"kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.804471 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.804448    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.804524 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.804503    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.864360 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.864333    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:39.866184 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.866156    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jj9\" (UniqueName: \"kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:39.923585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.922104    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"]
Oct 30 19:51:39.976904 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:39.976882    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:40.089646 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.089612    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"]
Oct 30 19:51:40.092718 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:51:40.092682    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd29001_8b89_4712_be5d_9700d0e864b5.slice/crio-5462363f8c3d0b7623eba610831d99bf6d203a75115fd5570f7f49e1bae0551c WatchSource:0}: Error finding container 5462363f8c3d0b7623eba610831d99bf6d203a75115fd5570f7f49e1bae0551c: Status 404 returned error can't find the container with id 5462363f8c3d0b7623eba610831d99bf6d203a75115fd5570f7f49e1bae0551c
Oct 30 19:51:40.237190 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.237126    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"]
Oct 30 19:51:40.246647 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:51:40.246622    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d9ffee_f4e9_40d3_a9a0_b8ced2190b70.slice/crio-7edb7f80f000755963142a1fde8eb3b8f2aa1876048688624f98ee4bc2f1e6d3 WatchSource:0}: Error finding container 7edb7f80f000755963142a1fde8eb3b8f2aa1876048688624f98ee4bc2f1e6d3: Status 404 returned error can't find the container with id 7edb7f80f000755963142a1fde8eb3b8f2aa1876048688624f98ee4bc2f1e6d3
Oct 30 19:51:40.352882 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.352859    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"]
Oct 30 19:51:40.411560 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:51:40.411533    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ad8960_dd5a_4ca5_9e87_e76f6312dca2.slice/crio-a66d94830abd2a652ee2125a71c5b6a507995fe9b28a3497cf5122da62a6fe7f WatchSource:0}: Error finding container a66d94830abd2a652ee2125a71c5b6a507995fe9b28a3497cf5122da62a6fe7f: Status 404 returned error can't find the container with id a66d94830abd2a652ee2125a71c5b6a507995fe9b28a3497cf5122da62a6fe7f
Oct 30 19:51:40.454389 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.454363    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb" event={"ID":"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2","Type":"ContainerStarted","Data":"a66d94830abd2a652ee2125a71c5b6a507995fe9b28a3497cf5122da62a6fe7f"}
Oct 30 19:51:40.455815 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.455794    4917 generic.go:356] "Generic (PLEG): container finished" podID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerID="b5e96d79c62a7fb5bd0ad67ddd0a8c2978b7a8c3f8384836d9a1ac92f5bba71f" exitCode=0
Oct 30 19:51:40.455921 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.455820    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4" event={"ID":"a3e63f13-0ef6-4bb0-a321-baf50ebaa982","Type":"ContainerDied","Data":"b5e96d79c62a7fb5bd0ad67ddd0a8c2978b7a8c3f8384836d9a1ac92f5bba71f"}
Oct 30 19:51:40.455921 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.455858    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4" event={"ID":"a3e63f13-0ef6-4bb0-a321-baf50ebaa982","Type":"ContainerStarted","Data":"3e971ac47f2b58801c505b31534e0b3f94b0505369bb921358f4afc9f440599c"}
Oct 30 19:51:40.457447 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.457416    4917 generic.go:356] "Generic (PLEG): container finished" podID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerID="207ae65553d70a6d7a2815f4ebd141e323a75964b5d335d629af4207156a43c9" exitCode=0
Oct 30 19:51:40.457556 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.457511    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw" event={"ID":"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70","Type":"ContainerDied","Data":"207ae65553d70a6d7a2815f4ebd141e323a75964b5d335d629af4207156a43c9"}
Oct 30 19:51:40.457556 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.457553    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw" event={"ID":"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70","Type":"ContainerStarted","Data":"7edb7f80f000755963142a1fde8eb3b8f2aa1876048688624f98ee4bc2f1e6d3"}
Oct 30 19:51:40.459122 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.459097    4917 generic.go:356] "Generic (PLEG): container finished" podID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerID="3cc99e766b000731056ee27d090a2a07c7f0e426547ca9511492f367d618ee9c" exitCode=0
Oct 30 19:51:40.459207 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.459149    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p" event={"ID":"4cd29001-8b89-4712-be5d-9700d0e864b5","Type":"ContainerDied","Data":"3cc99e766b000731056ee27d090a2a07c7f0e426547ca9511492f367d618ee9c"}
Oct 30 19:51:40.459207 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:40.459187    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p" event={"ID":"4cd29001-8b89-4712-be5d-9700d0e864b5","Type":"ContainerStarted","Data":"5462363f8c3d0b7623eba610831d99bf6d203a75115fd5570f7f49e1bae0551c"}
Oct 30 19:51:41.465330 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:41.465294    4917 generic.go:356] "Generic (PLEG): container finished" podID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerID="2a3c222f1a1af230e084b358aae7a40a63aaa5c040c349e9c0aa98e280b10da2" exitCode=0
Oct 30 19:51:41.465709 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:41.465384    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb" event={"ID":"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2","Type":"ContainerDied","Data":"2a3c222f1a1af230e084b358aae7a40a63aaa5c040c349e9c0aa98e280b10da2"}
Oct 30 19:51:42.472527 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:42.472448    4917 generic.go:356] "Generic (PLEG): container finished" podID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerID="f81b3bb1b5f64a5596becfc2738b45e47ac0f683654e31d6929495a30afd21ef" exitCode=0
Oct 30 19:51:42.472914 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:42.472541    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p" event={"ID":"4cd29001-8b89-4712-be5d-9700d0e864b5","Type":"ContainerDied","Data":"f81b3bb1b5f64a5596becfc2738b45e47ac0f683654e31d6929495a30afd21ef"}
Oct 30 19:51:43.480008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:43.479973    4917 generic.go:356] "Generic (PLEG): container finished" podID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerID="8a66af99f9d6844b4b4d51c0559c0c608e6d056bd08b19948b88971b4a3c253b" exitCode=0
Oct 30 19:51:43.480478 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:43.480055    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4" event={"ID":"a3e63f13-0ef6-4bb0-a321-baf50ebaa982","Type":"ContainerDied","Data":"8a66af99f9d6844b4b4d51c0559c0c608e6d056bd08b19948b88971b4a3c253b"}
Oct 30 19:51:43.481989 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:43.481969    4917 generic.go:356] "Generic (PLEG): container finished" podID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerID="7a13c5bf4e63ab46b702b526248d3524d585fd6bebac8cb5dde9641946a678c0" exitCode=0
Oct 30 19:51:43.482091 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:43.482045    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw" event={"ID":"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70","Type":"ContainerDied","Data":"7a13c5bf4e63ab46b702b526248d3524d585fd6bebac8cb5dde9641946a678c0"}
Oct 30 19:51:43.484608 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:43.484583    4917 generic.go:356] "Generic (PLEG): container finished" podID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerID="7e815df77e32fce2ca571acc06074fe4274dfaff37a853978f24e56847e78b27" exitCode=0
Oct 30 19:51:43.484715 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:43.484628    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p" event={"ID":"4cd29001-8b89-4712-be5d-9700d0e864b5","Type":"ContainerDied","Data":"7e815df77e32fce2ca571acc06074fe4274dfaff37a853978f24e56847e78b27"}
Oct 30 19:51:44.036019 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.035997    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:51:44.036147 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.036095    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:51:44.490626 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.490584    4917 generic.go:356] "Generic (PLEG): container finished" podID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerID="68723a33dd65a4a72ccec5672f6e8192f76c3c972e3d65acb86f50f8a11cdb07" exitCode=0
Oct 30 19:51:44.491107 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.490668    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb" event={"ID":"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2","Type":"ContainerDied","Data":"68723a33dd65a4a72ccec5672f6e8192f76c3c972e3d65acb86f50f8a11cdb07"}
Oct 30 19:51:44.493351 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.493328    4917 generic.go:356] "Generic (PLEG): container finished" podID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerID="06f3d09117274a632b95c4462c894a4c5f6d42dc28cd00e4f43275d09d87eefd" exitCode=0
Oct 30 19:51:44.493439 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.493419    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4" event={"ID":"a3e63f13-0ef6-4bb0-a321-baf50ebaa982","Type":"ContainerDied","Data":"06f3d09117274a632b95c4462c894a4c5f6d42dc28cd00e4f43275d09d87eefd"}
Oct 30 19:51:44.495530 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.495510    4917 generic.go:356] "Generic (PLEG): container finished" podID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerID="ca3dedb23c0ad3d1220d272ee8280015fac7cad66fae54f754bb1ef0ff5f34e6" exitCode=0
Oct 30 19:51:44.495616 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.495565    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw" event={"ID":"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70","Type":"ContainerDied","Data":"ca3dedb23c0ad3d1220d272ee8280015fac7cad66fae54f754bb1ef0ff5f34e6"}
Oct 30 19:51:44.638949 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.638924    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:44.744809 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.744702    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util\") pod \"4cd29001-8b89-4712-be5d-9700d0e864b5\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") "
Oct 30 19:51:44.744955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.744849    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f\" (UniqueName: \"kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f\") pod \"4cd29001-8b89-4712-be5d-9700d0e864b5\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") "
Oct 30 19:51:44.744955 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.744873    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle\") pod \"4cd29001-8b89-4712-be5d-9700d0e864b5\" (UID: \"4cd29001-8b89-4712-be5d-9700d0e864b5\") "
Oct 30 19:51:44.745500 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.745475    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle" pod "4cd29001-8b89-4712-be5d-9700d0e864b5" (UID: "4cd29001-8b89-4712-be5d-9700d0e864b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:44.747031 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.747006    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f" pod "4cd29001-8b89-4712-be5d-9700d0e864b5" (UID: "4cd29001-8b89-4712-be5d-9700d0e864b5"). InnerVolumeSpecName "kube-api-access-xmk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:44.750451 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.750426    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util" pod "4cd29001-8b89-4712-be5d-9700d0e864b5" (UID: "4cd29001-8b89-4712-be5d-9700d0e864b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:44.846455 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.846422    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:44.846455 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.846449    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmk5f\" (UniqueName: \"kubernetes.io/projected/4cd29001-8b89-4712-be5d-9700d0e864b5-kube-api-access-xmk5f\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:44.846455 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:44.846460    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd29001-8b89-4712-be5d-9700d0e864b5-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:45.502309 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.502236    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p"
Oct 30 19:51:45.502309 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.502240    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahzx2p" event={"ID":"4cd29001-8b89-4712-be5d-9700d0e864b5","Type":"ContainerDied","Data":"5462363f8c3d0b7623eba610831d99bf6d203a75115fd5570f7f49e1bae0551c"}
Oct 30 19:51:45.502309 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.502300    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5462363f8c3d0b7623eba610831d99bf6d203a75115fd5570f7f49e1bae0551c"
Oct 30 19:51:45.504409 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.504381    4917 generic.go:356] "Generic (PLEG): container finished" podID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerID="4441245d4a2a68ee5a3ac469bd83cb32162382c49ea748f7d7431e557ef888ad" exitCode=0
Oct 30 19:51:45.504543 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.504419    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb" event={"ID":"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2","Type":"ContainerDied","Data":"4441245d4a2a68ee5a3ac469bd83cb32162382c49ea748f7d7431e557ef888ad"}
Oct 30 19:51:45.637336 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.637312    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:45.652903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.652881    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle\") pod \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") "
Oct 30 19:51:45.653016 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.652942    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74\" (UniqueName: \"kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74\") pod \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") "
Oct 30 19:51:45.653016 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.652960    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util\") pod \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\" (UID: \"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70\") "
Oct 30 19:51:45.653452 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.653426    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle" pod "d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" (UID: "d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:45.654999 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.654979    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74" pod "d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" (UID: "d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70"). InnerVolumeSpecName "kube-api-access-9mn74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:45.665512 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.665489    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:45.753898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.753832    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle\") pod \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") "
Oct 30 19:51:45.753898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.753892    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util\") pod \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") "
Oct 30 19:51:45.754110 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.753921    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz\" (UniqueName: \"kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz\") pod \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\" (UID: \"a3e63f13-0ef6-4bb0-a321-baf50ebaa982\") "
Oct 30 19:51:45.754110 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.754080    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:45.754110 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.754097    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mn74\" (UniqueName: \"kubernetes.io/projected/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-kube-api-access-9mn74\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:45.754288 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.754265    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle" pod "a3e63f13-0ef6-4bb0-a321-baf50ebaa982" (UID: "a3e63f13-0ef6-4bb0-a321-baf50ebaa982"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:45.755933 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.755915    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz" pod "a3e63f13-0ef6-4bb0-a321-baf50ebaa982" (UID: "a3e63f13-0ef6-4bb0-a321-baf50ebaa982"). InnerVolumeSpecName "kube-api-access-vb8fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:45.807674 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.807640    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util" pod "d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" (UID: "d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:45.813690 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.813658    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util" pod "a3e63f13-0ef6-4bb0-a321-baf50ebaa982" (UID: "a3e63f13-0ef6-4bb0-a321-baf50ebaa982"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:45.854489 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.854457    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:45.854489 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.854484    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:45.854632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.854498    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8fz\" (UniqueName: \"kubernetes.io/projected/a3e63f13-0ef6-4bb0-a321-baf50ebaa982-kube-api-access-vb8fz\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:45.854632 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:45.854511    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:46.510904 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.510871    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4"
Oct 30 19:51:46.510904 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.510874    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q7kn4" event={"ID":"a3e63f13-0ef6-4bb0-a321-baf50ebaa982","Type":"ContainerDied","Data":"3e971ac47f2b58801c505b31534e0b3f94b0505369bb921358f4afc9f440599c"}
Oct 30 19:51:46.511283 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.510962    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e971ac47f2b58801c505b31534e0b3f94b0505369bb921358f4afc9f440599c"
Oct 30 19:51:46.513144 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.513110    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw" event={"ID":"d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70","Type":"ContainerDied","Data":"7edb7f80f000755963142a1fde8eb3b8f2aa1876048688624f98ee4bc2f1e6d3"}
Oct 30 19:51:46.513282 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.513183    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7edb7f80f000755963142a1fde8eb3b8f2aa1876048688624f98ee4bc2f1e6d3"
Oct 30 19:51:46.513322 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.513293    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhxgw"
Oct 30 19:51:46.640309 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.640288    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:51:46.660391 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.660367    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle\") pod \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") "
Oct 30 19:51:46.660484 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.660451    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9\" (UniqueName: \"kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9\") pod \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") "
Oct 30 19:51:46.660522 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.660506    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util\") pod \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\" (UID: \"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2\") "
Oct 30 19:51:46.660808 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.660759    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle" pod "b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" (UID: "b0ad8960-dd5a-4ca5-9e87-e76f6312dca2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:46.662520 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.662499    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9" pod "b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" (UID: "b0ad8960-dd5a-4ca5-9e87-e76f6312dca2"). InnerVolumeSpecName "kube-api-access-w2jj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:46.666667 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.666634    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util" pod "b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" (UID: "b0ad8960-dd5a-4ca5-9e87-e76f6312dca2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:46.761197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.761125    4917 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-bundle\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:46.761197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.761150    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jj9\" (UniqueName: \"kubernetes.io/projected/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-kube-api-access-w2jj9\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:46.761197 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:46.761159    4917 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0ad8960-dd5a-4ca5-9e87-e76f6312dca2-util\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:47.519883 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:47.519851    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb" event={"ID":"b0ad8960-dd5a-4ca5-9e87-e76f6312dca2","Type":"ContainerDied","Data":"a66d94830abd2a652ee2125a71c5b6a507995fe9b28a3497cf5122da62a6fe7f"}
Oct 30 19:51:47.520266 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:47.519906    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a66d94830abd2a652ee2125a71c5b6a507995fe9b28a3497cf5122da62a6fe7f"
Oct 30 19:51:47.520266 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:51:47.519950    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fl6sb"
Oct 30 19:52:09.678792 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:52:09.678744    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:52:18.522066 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522031    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"]
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522393    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522408    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522420    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522426    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522433    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522438    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522446    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522451    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522457    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522463    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522475    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522483    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522495    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522502    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522512    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522518    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522527    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522535    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="pull"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522541    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522545    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522555    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522559    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="util"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522565    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="extract"
Oct 30 19:52:18.522565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522569    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="extract"
Oct 30 19:52:18.523376 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522669    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd29001-8b89-4712-be5d-9700d0e864b5" containerName="extract"
Oct 30 19:52:18.523376 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522680    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d9ffee-f4e9-40d3-a9a0-b8ced2190b70" containerName="extract"
Oct 30 19:52:18.523376 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522686    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ad8960-dd5a-4ca5-9e87-e76f6312dca2" containerName="extract"
Oct 30 19:52:18.523376 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.522693    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e63f13-0ef6-4bb0-a321-baf50ebaa982" containerName="extract"
Oct 30 19:52:18.526960 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.526943    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.630265 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.630230    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6809a5a-77c6-46e1-9738-97f6d969b2ac-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.630439 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.630300    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b289m\" (UniqueName: \"kubernetes.io/projected/e6809a5a-77c6-46e1-9738-97f6d969b2ac-kube-api-access-b289m\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.630493 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.630471    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6809a5a-77c6-46e1-9738-97f6d969b2ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.632411 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.632391    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:52:18.632550 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.632414    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s9c2h\""
Oct 30 19:52:18.632550 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.632414    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 19:52:18.632550 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.632425    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 19:52:18.632550 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.632500    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:52:18.634406 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.634389    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"]
Oct 30 19:52:18.731380 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.731343    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6809a5a-77c6-46e1-9738-97f6d969b2ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.731555 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.731395    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6809a5a-77c6-46e1-9738-97f6d969b2ac-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.731555 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.731446    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b289m\" (UniqueName: \"kubernetes.io/projected/e6809a5a-77c6-46e1-9738-97f6d969b2ac-kube-api-access-b289m\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.732176 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.732156    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6809a5a-77c6-46e1-9738-97f6d969b2ac-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.733819 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.733789    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6809a5a-77c6-46e1-9738-97f6d969b2ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.793373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.793307    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b289m\" (UniqueName: \"kubernetes.io/projected/e6809a5a-77c6-46e1-9738-97f6d969b2ac-kube-api-access-b289m\") pod \"kuadrant-console-plugin-66c4fb575d-sddkh\" (UID: \"e6809a5a-77c6-46e1-9738-97f6d969b2ac\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:18.838007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:18.837978    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"
Oct 30 19:52:19.229597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:19.229569    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh"]
Oct 30 19:52:19.231530 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:52:19.231500    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6809a5a_77c6_46e1_9738_97f6d969b2ac.slice/crio-188f77c3d7bef58ef053304bdf2e1d23ca2c9745252d838890d01f0b04f5693d WatchSource:0}: Error finding container 188f77c3d7bef58ef053304bdf2e1d23ca2c9745252d838890d01f0b04f5693d: Status 404 returned error can't find the container with id 188f77c3d7bef58ef053304bdf2e1d23ca2c9745252d838890d01f0b04f5693d
Oct 30 19:52:19.679263 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:19.679229    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh" event={"ID":"e6809a5a-77c6-46e1-9738-97f6d969b2ac","Type":"ContainerStarted","Data":"188f77c3d7bef58ef053304bdf2e1d23ca2c9745252d838890d01f0b04f5693d"}
Oct 30 19:52:25.714226 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:25.714188    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh" event={"ID":"e6809a5a-77c6-46e1-9738-97f6d969b2ac","Type":"ContainerStarted","Data":"216c4983f491e550ca75884f4a7472731af1cf5d72cd63ac03df01f4ec5d579c"}
Oct 30 19:52:25.877802 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:25.877730    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sddkh" podStartSLOduration=2.087189406 podStartE2EDuration="7.877714663s" totalImagesPullingTime="5.790525257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:25.87590187 +0000 UTC m=+642.040303514" watchObservedRunningTime="2026-10-30 19:52:25.877714663 +0000 UTC m=+642.042116296"
Oct 30 19:52:39.679029 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:52:39.678986    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:52:59.694009 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.693933    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"]
Oct 30 19:52:59.697255 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.697236    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.780987 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.780953    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.780987 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.780990    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781247 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781030    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781247 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781115    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpllc\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781247 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781170    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781247 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781205    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-66kfh\""
Oct 30 19:52:59.781247 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781207    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781452 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781251    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781452 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781290    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.781452 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.781323    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.809320 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.809289    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"]
Oct 30 19:52:59.881809 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.881751    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.881809 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.881813    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.881866    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.881885    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpllc\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.881911    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882070    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882113    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882148    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882198    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882437 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882290    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882437 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882346    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882580 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882558    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882721 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882697    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.882878 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.882800    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.884218 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.884187    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.884502 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.884485    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.943691 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.943658    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:52:59.943926 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:52:59.943902    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpllc\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d49n\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:53:00.009731 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.009643    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:53:00.438907 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.438884    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"]
Oct 30 19:53:00.440578 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:53:00.440553    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377bbb6e_4fb5_47b4_be12_995d2dd14f89.slice/crio-bcd104197b097c31e95d2c30891eaf4cb4f9ed9d6d639e7cf8e946bbbf335fc9 WatchSource:0}: Error finding container bcd104197b097c31e95d2c30891eaf4cb4f9ed9d6d639e7cf8e946bbbf335fc9: Status 404 returned error can't find the container with id bcd104197b097c31e95d2c30891eaf4cb4f9ed9d6d639e7cf8e946bbbf335fc9
Oct 30 19:53:00.442594 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.442557    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:53:00.442674 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.442627    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:53:00.442674 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.442661    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:53:00.876839 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.876718    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" event={"ID":"377bbb6e-4fb5-47b4-be12-995d2dd14f89","Type":"ContainerStarted","Data":"2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"}
Oct 30 19:53:00.876839 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:00.876831    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" event={"ID":"377bbb6e-4fb5-47b4-be12-995d2dd14f89","Type":"ContainerStarted","Data":"bcd104197b097c31e95d2c30891eaf4cb4f9ed9d6d639e7cf8e946bbbf335fc9"}
Oct 30 19:53:01.003142 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:01.003092    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podStartSLOduration=2.003078885 podStartE2EDuration="2.003078885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:01.002325988 +0000 UTC m=+677.166727632" watchObservedRunningTime="2026-10-30 19:53:01.003078885 +0000 UTC m=+677.167480527"
Oct 30 19:53:01.010679 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:01.010648    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:53:01.015530 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:01.015505    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:53:01.882357 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:01.882326    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:53:01.883461 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:01.883443    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:53:09.678309 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:53:09.678265    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:53:19.439671 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.439635    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dzfbm"]
Oct 30 19:53:19.443621 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.443593    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:53:19.445764 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.445740    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:53:19.449047 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.449030    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.498315 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.498257    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5xpz2\""
Oct 30 19:53:19.500597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.500570    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dzfbm"]
Oct 30 19:53:19.502085 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.502063    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:53:19.550219 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.550182    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:53:19.562839 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.562808    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvkv\" (UniqueName: \"kubernetes.io/projected/9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56-kube-api-access-nhvkv\") pod \"limitador-limitador-7c9b87d574-gkr62\" (UID: \"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.562999 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.562878    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczpg\" (UniqueName: \"kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg\") pod \"authorino-56c6c9ddd5-dzfbm\" (UID: \"836bd9f1-6846-4e07-bfac-415359a04e56\") " pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:53:19.562999 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.562973    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56-config-file\") pod \"limitador-limitador-7c9b87d574-gkr62\" (UID: \"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.663879 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.663842    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczpg\" (UniqueName: \"kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg\") pod \"authorino-56c6c9ddd5-dzfbm\" (UID: \"836bd9f1-6846-4e07-bfac-415359a04e56\") " pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:53:19.664057 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.663887    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56-config-file\") pod \"limitador-limitador-7c9b87d574-gkr62\" (UID: \"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.664057 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.663941    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvkv\" (UniqueName: \"kubernetes.io/projected/9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56-kube-api-access-nhvkv\") pod \"limitador-limitador-7c9b87d574-gkr62\" (UID: \"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.664561 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.664540    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56-config-file\") pod \"limitador-limitador-7c9b87d574-gkr62\" (UID: \"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.726275 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.726190    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvkv\" (UniqueName: \"kubernetes.io/projected/9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56-kube-api-access-nhvkv\") pod \"limitador-limitador-7c9b87d574-gkr62\" (UID: \"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:19.726429 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.726333    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczpg\" (UniqueName: \"kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg\") pod \"authorino-56c6c9ddd5-dzfbm\" (UID: \"836bd9f1-6846-4e07-bfac-415359a04e56\") " pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:53:19.753861 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.753821    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:53:19.757798 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.757758    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:53:19.763816 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:19.763789    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:20.131931 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:20.131904    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dzfbm"]
Oct 30 19:53:20.133649 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:53:20.133620    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod836bd9f1_6846_4e07_bfac_415359a04e56.slice/crio-a9d8e8e84fc438c4bf971c9f0b1725013b74ca2a7aa4147ddae4718b084ae82b WatchSource:0}: Error finding container a9d8e8e84fc438c4bf971c9f0b1725013b74ca2a7aa4147ddae4718b084ae82b: Status 404 returned error can't find the container with id a9d8e8e84fc438c4bf971c9f0b1725013b74ca2a7aa4147ddae4718b084ae82b
Oct 30 19:53:20.135091 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:20.135075    4917 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:53:20.183612 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:20.183586    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:53:20.185521 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:53:20.185490    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9b5e5e_f34e_4513_98f0_a6f79a1f3f56.slice/crio-5f49ba479f16ca5e2a14101c91d85f40728c5efaf3b1b4d795f9fe0f1de0feac WatchSource:0}: Error finding container 5f49ba479f16ca5e2a14101c91d85f40728c5efaf3b1b4d795f9fe0f1de0feac: Status 404 returned error can't find the container with id 5f49ba479f16ca5e2a14101c91d85f40728c5efaf3b1b4d795f9fe0f1de0feac
Oct 30 19:53:20.973154 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:20.973114    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62" event={"ID":"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56","Type":"ContainerStarted","Data":"5f49ba479f16ca5e2a14101c91d85f40728c5efaf3b1b4d795f9fe0f1de0feac"}
Oct 30 19:53:20.974281 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:20.974260    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm" event={"ID":"836bd9f1-6846-4e07-bfac-415359a04e56","Type":"ContainerStarted","Data":"a9d8e8e84fc438c4bf971c9f0b1725013b74ca2a7aa4147ddae4718b084ae82b"}
Oct 30 19:53:24.001808 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:24.001757    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62" event={"ID":"9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56","Type":"ContainerStarted","Data":"96402ada241687195ca95f411de3a56af6c3fc88aaaf1b94ac75687cb7e565fe"}
Oct 30 19:53:24.002133 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:24.001847    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:24.117371 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:24.117269    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62" podStartSLOduration=1.441421308 podStartE2EDuration="5.117256668s" totalImagesPullingTime="3.67583536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:24.116400484 +0000 UTC m=+700.280802125" watchObservedRunningTime="2026-10-30 19:53:24.117256668 +0000 UTC m=+700.281658352"
Oct 30 19:53:25.004612 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:25.004568    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm" event={"ID":"836bd9f1-6846-4e07-bfac-415359a04e56","Type":"ContainerStarted","Data":"809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361"}
Oct 30 19:53:25.120204 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:25.120151    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm" podStartSLOduration=2.260915624 podStartE2EDuration="6.120138121s" totalImagesPullingTime="3.859222497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:25.119888273 +0000 UTC m=+701.284289916" watchObservedRunningTime="2026-10-30 19:53:25.120138121 +0000 UTC m=+701.284539760"
Oct 30 19:53:35.005912 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:35.005877    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkr62"
Oct 30 19:53:39.678635 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:53:39.678592    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:53:59.349505 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:59.349470    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dzfbm"]
Oct 30 19:53:59.350068 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:59.349674    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm" podUID="836bd9f1-6846-4e07-bfac-415359a04e56" containerName="authorino" containerID="cri-o://809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361" gracePeriod=30
Oct 30 19:53:59.595759 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:59.595734    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:53:59.700386 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:59.700356    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg\" (UniqueName: \"kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg\") pod \"836bd9f1-6846-4e07-bfac-415359a04e56\" (UID: \"836bd9f1-6846-4e07-bfac-415359a04e56\") "
Oct 30 19:53:59.702598 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:59.702566    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg" pod "836bd9f1-6846-4e07-bfac-415359a04e56" (UID: "836bd9f1-6846-4e07-bfac-415359a04e56"). InnerVolumeSpecName "kube-api-access-kczpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:59.801224 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:53:59.801188    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kczpg\" (UniqueName: \"kubernetes.io/projected/836bd9f1-6846-4e07-bfac-415359a04e56-kube-api-access-kczpg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:00.172663 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.172626    4917 generic.go:356] "Generic (PLEG): container finished" podID="836bd9f1-6846-4e07-bfac-415359a04e56" containerID="809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361" exitCode=0
Oct 30 19:54:00.172883 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.172671    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm"
Oct 30 19:54:00.172883 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.172679    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm" event={"ID":"836bd9f1-6846-4e07-bfac-415359a04e56","Type":"ContainerDied","Data":"809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361"}
Oct 30 19:54:00.172883 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.172733    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dzfbm" event={"ID":"836bd9f1-6846-4e07-bfac-415359a04e56","Type":"ContainerDied","Data":"a9d8e8e84fc438c4bf971c9f0b1725013b74ca2a7aa4147ddae4718b084ae82b"}
Oct 30 19:54:00.172883 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.172756    4917 scope.go:122] "RemoveContainer" containerID="809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361"
Oct 30 19:54:00.181622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.181607    4917 scope.go:122] "RemoveContainer" containerID="809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361"
Oct 30 19:54:00.181909 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:54:00.181890    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361\": container with ID starting with 809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361 not found: ID does not exist" containerID="809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361"
Oct 30 19:54:00.181979 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.181917    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361"} err="failed to get container status \"809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361\": rpc error: code = NotFound desc = could not find container \"809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361\": container with ID starting with 809f54e37bb20e3a71af9e1b987448e9370fb1d515008693c916aa7f30162361 not found: ID does not exist"
Oct 30 19:54:00.351693 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.351614    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dzfbm"]
Oct 30 19:54:00.353359 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:00.353338    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dzfbm"]
Oct 30 19:54:02.003944 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:02.001946    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836bd9f1-6846-4e07-bfac-415359a04e56" path="/var/lib/kubelet/pods/836bd9f1-6846-4e07-bfac-415359a04e56/volumes"
Oct 30 19:54:09.678587 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:54:09.678543    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:54:39.679035 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:54:39.678956    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:54:52.685037 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:52.685000    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:54:57.344796 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.341133    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"]
Oct 30 19:54:57.363479 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.363438    4917 cpu_manager.go:414] "containerMap: removing container" podUID="836bd9f1-6846-4e07-bfac-415359a04e56" containerName="authorino"
Oct 30 19:54:57.363716 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.363698    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836bd9f1-6846-4e07-bfac-415359a04e56" containerName="authorino"
Oct 30 19:54:57.363951 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.363935    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="836bd9f1-6846-4e07-bfac-415359a04e56" containerName="authorino"
Oct 30 19:54:57.368597 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.368572    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.403914 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.403883    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"]
Oct 30 19:54:57.407354 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.407327    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eec900b3-e382-413e-aa96-c42ca854041d-cert\") pod \"llmisvc-controller-manager-6466c554d8-5pdj6\" (UID: \"eec900b3-e382-413e-aa96-c42ca854041d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.407479 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.407401    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjwz\" (UniqueName: \"kubernetes.io/projected/eec900b3-e382-413e-aa96-c42ca854041d-kube-api-access-dgjwz\") pod \"llmisvc-controller-manager-6466c554d8-5pdj6\" (UID: \"eec900b3-e382-413e-aa96-c42ca854041d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.508363 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.508324    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjwz\" (UniqueName: \"kubernetes.io/projected/eec900b3-e382-413e-aa96-c42ca854041d-kube-api-access-dgjwz\") pod \"llmisvc-controller-manager-6466c554d8-5pdj6\" (UID: \"eec900b3-e382-413e-aa96-c42ca854041d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.508574 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.508392    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eec900b3-e382-413e-aa96-c42ca854041d-cert\") pod \"llmisvc-controller-manager-6466c554d8-5pdj6\" (UID: \"eec900b3-e382-413e-aa96-c42ca854041d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.510837 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.510813    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eec900b3-e382-413e-aa96-c42ca854041d-cert\") pod \"llmisvc-controller-manager-6466c554d8-5pdj6\" (UID: \"eec900b3-e382-413e-aa96-c42ca854041d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.570799 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.570738    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjwz\" (UniqueName: \"kubernetes.io/projected/eec900b3-e382-413e-aa96-c42ca854041d-kube-api-access-dgjwz\") pod \"llmisvc-controller-manager-6466c554d8-5pdj6\" (UID: \"eec900b3-e382-413e-aa96-c42ca854041d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:57.682248 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:57.682208    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:58.066484 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:58.066398    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"]
Oct 30 19:54:58.070210 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:54:58.070183    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec900b3_e382_413e_aa96_c42ca854041d.slice/crio-29ed929e25e37b6f144458a32ed380205869ff1b89c24a377992c8c8ddfb966f WatchSource:0}: Error finding container 29ed929e25e37b6f144458a32ed380205869ff1b89c24a377992c8c8ddfb966f: Status 404 returned error can't find the container with id 29ed929e25e37b6f144458a32ed380205869ff1b89c24a377992c8c8ddfb966f
Oct 30 19:54:58.481321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:58.481284    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6" event={"ID":"eec900b3-e382-413e-aa96-c42ca854041d","Type":"ContainerStarted","Data":"29ed929e25e37b6f144458a32ed380205869ff1b89c24a377992c8c8ddfb966f"}
Oct 30 19:54:59.488310 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:59.488262    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6" event={"ID":"eec900b3-e382-413e-aa96-c42ca854041d","Type":"ContainerStarted","Data":"4307b37fb3c90ff6140ba749be18f566f4c984841c8ed1b0231709308b8b4bda"}
Oct 30 19:54:59.488734 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:59.488378    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:54:59.615006 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:54:59.614947    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6" podStartSLOduration=1.8880166379999999 podStartE2EDuration="2.614933083s" totalImagesPullingTime="726.916445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:59.613228851 +0000 UTC m=+795.777630507" watchObservedRunningTime="2026-10-30 19:54:59.614933083 +0000 UTC m=+795.779334725"
Oct 30 19:55:01.883842 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:01.883810    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d49n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:55:01.884246 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:01.883876    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:55:02.503845 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:02.503813    4917 generic.go:356] "Generic (PLEG): container finished" podID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerID="2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92" exitCode=0
Oct 30 19:55:02.504023 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:02.503891    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" event={"ID":"377bbb6e-4fb5-47b4-be12-995d2dd14f89","Type":"ContainerDied","Data":"2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"}
Oct 30 19:55:02.504276 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:02.504262    4917 scope.go:122] "RemoveContainer" containerID="2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"
Oct 30 19:55:02.505334 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:02.505309    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:02.505391 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:02.505373    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:02.505424 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:02.505403    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:55:03.517387 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:03.517347    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" event={"ID":"377bbb6e-4fb5-47b4-be12-995d2dd14f89","Type":"ContainerStarted","Data":"4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770"}
Oct 30 19:55:04.010237 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:04.010203    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:55:04.010426 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:04.010404    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:55:04.010475 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:04.010460    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:55:05.010759 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:05.010720    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:55:05.011282 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:05.010811    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:55:06.010707 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:06.010672    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:55:06.010898 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:06.010736    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:55:07.011114 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:07.011073    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:55:07.011524 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:07.011128    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:55:08.013198 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:08.013157    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:55:08.013786 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:08.013738    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:55:09.014194 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:09.014160    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:55:09.014627 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:09.014459    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:55:09.015164 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:09.015146    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 19:55:09.679194 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:55:09.679158    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:55:30.495696 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:30.495662    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5pdj6"
Oct 30 19:55:30.667621 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:30.667587    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"]
Oct 30 19:55:30.667892 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:30.667866    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z" podUID="80317fab-980b-49a3-97eb-1ea5a577d79d" containerName="manager" containerID="cri-o://b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8" gracePeriod=30
Oct 30 19:55:30.920814 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:30.920781    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:55:31.017842 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.017809    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7\" (UniqueName: \"kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7\") pod \"80317fab-980b-49a3-97eb-1ea5a577d79d\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") "
Oct 30 19:55:31.018030 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.017849    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert\" (UniqueName: \"kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert\") pod \"80317fab-980b-49a3-97eb-1ea5a577d79d\" (UID: \"80317fab-980b-49a3-97eb-1ea5a577d79d\") "
Oct 30 19:55:31.019967 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.019942    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert" pod "80317fab-980b-49a3-97eb-1ea5a577d79d" (UID: "80317fab-980b-49a3-97eb-1ea5a577d79d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:31.020065 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.020010    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7" pod "80317fab-980b-49a3-97eb-1ea5a577d79d" (UID: "80317fab-980b-49a3-97eb-1ea5a577d79d"). InnerVolumeSpecName "kube-api-access-tl6h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:31.119058 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.118967    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6h7\" (UniqueName: \"kubernetes.io/projected/80317fab-980b-49a3-97eb-1ea5a577d79d-kube-api-access-tl6h7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:31.119058 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.119000    4917 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80317fab-980b-49a3-97eb-1ea5a577d79d-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:31.677803 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.677748    4917 generic.go:356] "Generic (PLEG): container finished" podID="80317fab-980b-49a3-97eb-1ea5a577d79d" containerID="b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8" exitCode=0
Oct 30 19:55:31.678252 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.677811    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"
Oct 30 19:55:31.678252 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.677833    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z" event={"ID":"80317fab-980b-49a3-97eb-1ea5a577d79d","Type":"ContainerDied","Data":"b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8"}
Oct 30 19:55:31.678252 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.677887    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z" event={"ID":"80317fab-980b-49a3-97eb-1ea5a577d79d","Type":"ContainerDied","Data":"254d7d6efe92dee63c6a26bab6112dfe99ae6af8d17a5741bb0053bd62fcaad5"}
Oct 30 19:55:31.678252 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.677908    4917 scope.go:122] "RemoveContainer" containerID="b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8"
Oct 30 19:55:31.686981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.686958    4917 scope.go:122] "RemoveContainer" containerID="b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8"
Oct 30 19:55:31.687262 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:55:31.687244    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8\": container with ID starting with b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8 not found: ID does not exist" containerID="b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8"
Oct 30 19:55:31.687321 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.687273    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8"} err="failed to get container status \"b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8\": rpc error: code = NotFound desc = could not find container \"b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8\": container with ID starting with b4e2315ea175b187fe2b7f6562b74c13060e5f801d1c3aca8105dddbd83fc8e8 not found: ID does not exist"
Oct 30 19:55:31.862835 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.862794    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"]
Oct 30 19:55:31.911920 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:31.911876    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8kc6z"]
Oct 30 19:55:32.001173 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:32.001097    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80317fab-980b-49a3-97eb-1ea5a577d79d" path="/var/lib/kubelet/pods/80317fab-980b-49a3-97eb-1ea5a577d79d/volumes"
Oct 30 19:55:39.679183 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:55:39.679145    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:55:51.987786 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:55:51.987733    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:56:01.928452 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:56:01.928367    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:56:09.679703 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:56:09.679660    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:56:18.822131 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:56:18.822092    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:56:39.679724 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:56:39.679682    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:56:44.084404 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:56:44.084374    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:56:44.086489 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:56:44.086467    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 19:57:09.678638 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:57:09.678601    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:57:39.679232 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:57:39.679152    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:58:01.488207 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:01.488170    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:58:03.607665 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.607634    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"]
Oct 30 19:58:03.608076 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.608032    4917 cpu_manager.go:414] "containerMap: removing container" podUID="80317fab-980b-49a3-97eb-1ea5a577d79d" containerName="manager"
Oct 30 19:58:03.608076 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.608045    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80317fab-980b-49a3-97eb-1ea5a577d79d" containerName="manager"
Oct 30 19:58:03.608153 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.608110    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="80317fab-980b-49a3-97eb-1ea5a577d79d" containerName="manager"
Oct 30 19:58:03.611222 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.611205    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:03.708636 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.708605    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-3ccc9f\""
Oct 30 19:58:03.708969 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.708934    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-3ccc9f-dockercfg-644jg\""
Oct 30 19:58:03.761438 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.761405    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"]
Oct 30 19:58:03.771518 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.771468    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:03.771699 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.771610    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qkcm\" (UniqueName: \"kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm\") pod \"payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:03.872885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.872791    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:03.872885 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.872869    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qkcm\" (UniqueName: \"kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm\") pod \"payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:03.873548 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.873520    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:03.935507 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:03.935476    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qkcm\" (UniqueName: \"kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm\") pod \"payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:04.224360 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:04.224321    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:04.597336 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:04.597312    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"]
Oct 30 19:58:04.599204 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:58:04.599175    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf57283a_9f6d_4e48_8977_15e91aa67e98.slice/crio-e1439a340284a30c98a4e0b3d2ff220d96071490330cb1317675d474c865a27d WatchSource:0}: Error finding container e1439a340284a30c98a4e0b3d2ff220d96071490330cb1317675d474c865a27d: Status 404 returned error can't find the container with id e1439a340284a30c98a4e0b3d2ff220d96071490330cb1317675d474c865a27d
Oct 30 19:58:05.441003 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:05.440961    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx" event={"ID":"af57283a-9f6d-4e48-8977-15e91aa67e98","Type":"ContainerStarted","Data":"e1439a340284a30c98a4e0b3d2ff220d96071490330cb1317675d474c865a27d"}
Oct 30 19:58:06.497110 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.497080    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"]
Oct 30 19:58:06.505519 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.504982    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.562076 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.562033    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f84d85\""
Oct 30 19:58:06.603456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.603235    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjck\" (UniqueName: \"kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck\") pod \"payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.603456 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.603327    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.612502 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.612427    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f84d85-dockercfg-knphb\""
Oct 30 19:58:06.614938 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.614904    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"]
Oct 30 19:58:06.638786 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.638745    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"]
Oct 30 19:58:06.645680 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.645655    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.701142 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.701104    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"]
Oct 30 19:58:06.704598 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.704569    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.704780 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.704613    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjck\" (UniqueName: \"kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck\") pod \"payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.704851 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.704818    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.704911 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.704898    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnng\" (UniqueName: \"kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng\") pod \"payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.705842 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.705814    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.767025 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.766941    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjck\" (UniqueName: \"kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck\") pod \"payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.805730 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.805697    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.805933 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.805830    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnng\" (UniqueName: \"kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng\") pod \"payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.806636 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.806612    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.823105 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.823077    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:06.876981 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.876947    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnng\" (UniqueName: \"kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng\") pod \"payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:06.962293 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:06.962253    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:07.217887 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:07.217855    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"]
Oct 30 19:58:07.219669 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:58:07.219642    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece176ef_2f7f_4348_aa65_82d1150fadf4.slice/crio-cc6ca0cf0bf5800f56aaa3d51b6977faa6fecd85266c7e99b596cb503fe6c5a9 WatchSource:0}: Error finding container cc6ca0cf0bf5800f56aaa3d51b6977faa6fecd85266c7e99b596cb503fe6c5a9: Status 404 returned error can't find the container with id cc6ca0cf0bf5800f56aaa3d51b6977faa6fecd85266c7e99b596cb503fe6c5a9
Oct 30 19:58:07.346738 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:07.346707    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"]
Oct 30 19:58:07.348620 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:58:07.348589    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab70257_8dab_41ea_8a28_2ecf2fc0e790.slice/crio-8963d2cb36169e929c9e362ab0dea7e8dd116161eab277b12c1d7d8f66684df2 WatchSource:0}: Error finding container 8963d2cb36169e929c9e362ab0dea7e8dd116161eab277b12c1d7d8f66684df2: Status 404 returned error can't find the container with id 8963d2cb36169e929c9e362ab0dea7e8dd116161eab277b12c1d7d8f66684df2
Oct 30 19:58:07.453171 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:07.453145    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" event={"ID":"ece176ef-2f7f-4348-aa65-82d1150fadf4","Type":"ContainerStarted","Data":"cc6ca0cf0bf5800f56aaa3d51b6977faa6fecd85266c7e99b596cb503fe6c5a9"}
Oct 30 19:58:07.454339 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:07.454310    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" event={"ID":"eab70257-8dab-41ea-8a28-2ecf2fc0e790","Type":"ContainerStarted","Data":"8963d2cb36169e929c9e362ab0dea7e8dd116161eab277b12c1d7d8f66684df2"}
Oct 30 19:58:08.460711 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.460663    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" event={"ID":"ece176ef-2f7f-4348-aa65-82d1150fadf4","Type":"ContainerStarted","Data":"6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b"}
Oct 30 19:58:08.461181 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.460763    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:08.462260 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.462236    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx" event={"ID":"af57283a-9f6d-4e48-8977-15e91aa67e98","Type":"ContainerStarted","Data":"226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55"}
Oct 30 19:58:08.462373 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.462293    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:08.463685 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.463664    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" event={"ID":"eab70257-8dab-41ea-8a28-2ecf2fc0e790","Type":"ContainerStarted","Data":"406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8"}
Oct 30 19:58:08.463810 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.463708    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:08.579548 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.579445    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" podStartSLOduration=1.790197607 podStartE2EDuration="2.579430633s" totalImagesPullingTime="789.233026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:08.578371178 +0000 UTC m=+984.742772822" watchObservedRunningTime="2026-10-30 19:58:08.579430633 +0000 UTC m=+984.743832274"
Oct 30 19:58:08.699843 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.699749    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx" podStartSLOduration=2.914980266 podStartE2EDuration="5.699729692s" totalImagesPullingTime="2.784749426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:08.699581306 +0000 UTC m=+984.863982949" watchObservedRunningTime="2026-10-30 19:58:08.699729692 +0000 UTC m=+984.864131335"
Oct 30 19:58:08.820537 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:08.820486    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" podStartSLOduration=1.968320196 podStartE2EDuration="2.820472022s" totalImagesPullingTime="852.151826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:08.819161108 +0000 UTC m=+984.983562750" watchObservedRunningTime="2026-10-30 19:58:08.820472022 +0000 UTC m=+984.984873664"
Oct 30 19:58:09.679004 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:58:09.678965    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:58:10.270315 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:10.270281    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:58:17.258173 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.258140    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"]
Oct 30 19:58:17.262883 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.262857    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.302609 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.302568    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.302798 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.302720    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6b8\" (UniqueName: \"kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8\") pod \"payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.319412 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.319384    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bebb42-dockercfg-c275n\""
Oct 30 19:58:17.319608 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.319399    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bebb42\""
Oct 30 19:58:17.321919 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.321897    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"]
Oct 30 19:58:17.383467 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.383427    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"]
Oct 30 19:58:17.387127 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.387101    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.403544 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.403514    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6b8\" (UniqueName: \"kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8\") pod \"payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.403667 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.403557    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.404235 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.404208    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.447183 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.447145    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"]
Oct 30 19:58:17.466156 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.466124    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6b8\" (UniqueName: \"kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8\") pod \"payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.504078 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.504041    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.504215 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.504185    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsq9\" (UniqueName: \"kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9\") pod \"payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.577008 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.576920    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:17.605421 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.605380    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.605585 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.605485    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsq9\" (UniqueName: \"kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9\") pod \"payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.606341 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.606317    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.668379 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.668315    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsq9\" (UniqueName: \"kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9\") pod \"payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.701165 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.701124    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:17.979555 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:17.979527    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"]
Oct 30 19:58:17.981373 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:58:17.981347    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a213d8a_0c63_4487_a624_1ae629078a20.slice/crio-d58406b60498acd0bfffabc2f3e814a7cbca7101ed8993fac5230ace0c0e46ac WatchSource:0}: Error finding container d58406b60498acd0bfffabc2f3e814a7cbca7101ed8993fac5230ace0c0e46ac: Status 404 returned error can't find the container with id d58406b60498acd0bfffabc2f3e814a7cbca7101ed8993fac5230ace0c0e46ac
Oct 30 19:58:18.074263 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.074238    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"]
Oct 30 19:58:18.076609 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:58:18.076585    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5bbf08_ce59_4b13_af66_297f73dab2d5.slice/crio-187191dade308425c69cba1c646ce99a98398a07efb86c3a03237e10b5fdbaaa WatchSource:0}: Error finding container 187191dade308425c69cba1c646ce99a98398a07efb86c3a03237e10b5fdbaaa: Status 404 returned error can't find the container with id 187191dade308425c69cba1c646ce99a98398a07efb86c3a03237e10b5fdbaaa
Oct 30 19:58:18.518188 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.518156    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" event={"ID":"1d5bbf08-ce59-4b13-af66-297f73dab2d5","Type":"ContainerStarted","Data":"f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb"}
Oct 30 19:58:18.518631 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.518221    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:18.518631 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.518248    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" event={"ID":"1d5bbf08-ce59-4b13-af66-297f73dab2d5","Type":"ContainerStarted","Data":"187191dade308425c69cba1c646ce99a98398a07efb86c3a03237e10b5fdbaaa"}
Oct 30 19:58:18.519738 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.519711    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" event={"ID":"2a213d8a-0c63-4487-a624-1ae629078a20","Type":"ContainerStarted","Data":"036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb"}
Oct 30 19:58:18.519903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.519749    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" event={"ID":"2a213d8a-0c63-4487-a624-1ae629078a20","Type":"ContainerStarted","Data":"d58406b60498acd0bfffabc2f3e814a7cbca7101ed8993fac5230ace0c0e46ac"}
Oct 30 19:58:18.519903 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.519799    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:18.636469 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.636407    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" podStartSLOduration=1.636393774 podStartE2EDuration="1.636393774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:18.634918336 +0000 UTC m=+994.799319979" watchObservedRunningTime="2026-10-30 19:58:18.636393774 +0000 UTC m=+994.800795416"
Oct 30 19:58:18.702803 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.702755    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:58:18.752968 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:18.752916    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" podStartSLOduration=1.752900133 podStartE2EDuration="1.752900133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:18.75206551 +0000 UTC m=+994.916467168" watchObservedRunningTime="2026-10-30 19:58:18.752900133 +0000 UTC m=+994.917301771"
Oct 30 19:58:19.471537 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:19.471505    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 19:58:19.471726 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:19.471698    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 19:58:19.471789 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:19.471749    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 19:58:29.528485 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:29.528454    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 19:58:29.529058 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:29.528535    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 19:58:31.219260 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:31.219213    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:58:39.678446 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:58:39.678405    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:58:39.850889 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:39.850855    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:58:46.609954 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.609916    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"]
Oct 30 19:58:46.617572 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.617547    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.674106 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.674073    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"]
Oct 30 19:58:46.722234 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.722208    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"default-dockercfg-jjd8c\""
Oct 30 19:58:46.722395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.722327    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"simulator-w5-407632-kserve-self-signed-certs\""
Oct 30 19:58:46.722395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.722336    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"kube-root-ca.crt\""
Oct 30 19:58:46.722395 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.722345    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"openshift-service-ca.crt\""
Oct 30 19:58:46.778222 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.778191    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.778393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.778236    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnfz\" (UniqueName: \"kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.778393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.778268    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.778393 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.778371    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.778500 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.778392    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.778500 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.778430    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.879829 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.879742    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.879829 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.879796    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880049 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.879854    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880049 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.879915    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880049 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.879959    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnfz\" (UniqueName: \"kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880236 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.880201    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880404 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.880382    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880560 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.880541    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.880758 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.880738    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.882457 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.882430    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.882565 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.882464    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:46.943414 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:46.943385    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnfz\" (UniqueName: \"kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz\") pod \"simulator-w5-407632-kserve-696559477b-ckrgp\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") " pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:47.229114 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:47.229081    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:47.602414 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:47.602392    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"]
Oct 30 19:58:47.604715 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:58:47.604693    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d2bbfa_e4ff_4489_a673_61d5aac0dadf.slice/crio-cb4eab443f970c437aaf6f37526e8e399f804046496f2bee4e0755ae25bdb662 WatchSource:0}: Error finding container cb4eab443f970c437aaf6f37526e8e399f804046496f2bee4e0755ae25bdb662: Status 404 returned error can't find the container with id cb4eab443f970c437aaf6f37526e8e399f804046496f2bee4e0755ae25bdb662
Oct 30 19:58:47.606581 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:47.606559    4917 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:58:47.674846 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:47.674811    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" event={"ID":"17d2bbfa-e4ff-4489-a673-61d5aac0dadf","Type":"ContainerStarted","Data":"cb4eab443f970c437aaf6f37526e8e399f804046496f2bee4e0755ae25bdb662"}
Oct 30 19:58:49.701875 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:49.701826    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" event={"ID":"17d2bbfa-e4ff-4489-a673-61d5aac0dadf","Type":"ContainerStarted","Data":"1bf00595c6ee1950569ef24cda6b6667c2d57917dc32afa0be62b9490dc9a5fe"}
Oct 30 19:58:49.826264 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:49.826208    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podStartSLOduration=2.110284007 podStartE2EDuration="3.826193606s" totalImagesPullingTime="1.715909599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:49.823981487 +0000 UTC m=+1025.988383131" watchObservedRunningTime="2026-10-30 19:58:49.826193606 +0000 UTC m=+1025.990595248"
Oct 30 19:58:50.229790 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:50.229747    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:53.733890 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:53.733859    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:58:57.229951 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:57.229918    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:57.242581 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:57.242551    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:58:57.757961 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:58:57.757931    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 19:59:09.678376 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:59:09.678338    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:59:13.860750 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:13.860704    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:59:15.768622 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.768586    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"]
Oct 30 19:59:15.772241 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.772209    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.832914 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.832885    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.833064 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.832939    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.833064 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.833015    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.833064 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.833051    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.833242 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.833156    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rpb\" (UniqueName: \"kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.833242 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.833181    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.876917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.876889    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"kube-root-ca.crt\""
Oct 30 19:59:15.876917 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.876885    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"openshift-service-ca.crt\""
Oct 30 19:59:15.877118 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.877105    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"default-dockercfg-z9mnr\""
Oct 30 19:59:15.877269 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.877255    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"simulator-w1-3ccc9f-kserve-self-signed-certs\""
Oct 30 19:59:15.879623 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.879605    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"]
Oct 30 19:59:15.933996 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.933964    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpb\" (UniqueName: \"kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.933996 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.933995    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934030    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934059    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934081    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934213 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934100    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934442 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934416    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934569 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934482    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.934569 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.934538    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.936297 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.936273    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.936552 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.936535    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:15.997482 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:15.997453    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rpb\" (UniqueName: \"kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb\") pod \"simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") " pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:16.084035 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:16.083947    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:16.466926 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:16.466901    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"]
Oct 30 19:59:16.468700 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:59:16.468674    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fae092c_192f_4272_9edc_d46a060463b3.slice/crio-6b837246018d99ba20c73ebbca3490cf7831e5370ef91579f18226df53f5ecb1 WatchSource:0}: Error finding container 6b837246018d99ba20c73ebbca3490cf7831e5370ef91579f18226df53f5ecb1: Status 404 returned error can't find the container with id 6b837246018d99ba20c73ebbca3490cf7831e5370ef91579f18226df53f5ecb1
Oct 30 19:59:16.856055 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:16.855965    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" event={"ID":"2fae092c-192f-4272-9edc-d46a060463b3","Type":"ContainerStarted","Data":"d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550"}
Oct 30 19:59:16.856055 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:16.856010    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" event={"ID":"2fae092c-192f-4272-9edc-d46a060463b3","Type":"ContainerStarted","Data":"6b837246018d99ba20c73ebbca3490cf7831e5370ef91579f18226df53f5ecb1"}
Oct 30 19:59:16.975607 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:16.975553    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podStartSLOduration=1.9755400829999998 podStartE2EDuration="1.975540083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:16.973957977 +0000 UTC m=+1053.138359620" watchObservedRunningTime="2026-10-30 19:59:16.975540083 +0000 UTC m=+1053.139941717"
Oct 30 19:59:17.084023 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:17.083996    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:26.084219 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:26.084176    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:26.096938 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:26.096911    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:26.920257 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:26.920224    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 19:59:30.890574 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:30.890537    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:59:38.260698 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.260655    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"]
Oct 30 19:59:38.264494 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.264470    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.338599 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.338557    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.338599 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.338597    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86v7\" (UniqueName: \"kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.338865 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.338718    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.338865 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.338784    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.338865 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.338841    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.338999 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.338871    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.367976 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.367950    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"kube-root-ca.crt\""
Oct 30 19:59:38.367976 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.367954    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"simulator-w2-d67ea9-kserve-self-signed-certs\""
Oct 30 19:59:38.368180 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.368059    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"openshift-service-ca.crt\""
Oct 30 19:59:38.368180 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.368143    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"default-dockercfg-q5s7s\""
Oct 30 19:59:38.370342 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.370322    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"]
Oct 30 19:59:38.439278 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439245    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.439464 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439287    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86v7\" (UniqueName: \"kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.439464 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439362    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.439464 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439406    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.439631 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439555    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.439631 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439584    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.439884 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439859    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.440016 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439929    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.440016 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.439957    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.442245 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.442226    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.442652 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.442631    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.501470 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.501438    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86v7\" (UniqueName: \"kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7\") pod \"simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") " pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.577243 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.577146    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:38.967496 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:38.967465    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"]
Oct 30 19:59:38.970511 ip-10-0-133-27 kubenswrapper[4917]: W1030 19:59:38.970480    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77350f60_57fa_40a5_8fe5_742c5b5f7979.slice/crio-b9dd41fd10f5904da06725a0d4edc9a21e3724e4f036a1780c5de508ea4f7633 WatchSource:0}: Error finding container b9dd41fd10f5904da06725a0d4edc9a21e3724e4f036a1780c5de508ea4f7633: Status 404 returned error can't find the container with id b9dd41fd10f5904da06725a0d4edc9a21e3724e4f036a1780c5de508ea4f7633
Oct 30 19:59:39.679159 ip-10-0-133-27 kubenswrapper[4917]: E1030 19:59:39.679126    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 19:59:39.981904 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:39.981809    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" event={"ID":"77350f60-57fa-40a5-8fe5-742c5b5f7979","Type":"ContainerStarted","Data":"6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515"}
Oct 30 19:59:39.981904 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:39.981860    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" event={"ID":"77350f60-57fa-40a5-8fe5-742c5b5f7979","Type":"ContainerStarted","Data":"b9dd41fd10f5904da06725a0d4edc9a21e3724e4f036a1780c5de508ea4f7633"}
Oct 30 19:59:40.105553 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:40.105504    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podStartSLOduration=2.105489378 podStartE2EDuration="2.105489378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:40.104323173 +0000 UTC m=+1076.268724826" watchObservedRunningTime="2026-10-30 19:59:40.105489378 +0000 UTC m=+1076.269891020"
Oct 30 19:59:40.577787 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:40.577734    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:45.812688 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:45.812654    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 19:59:48.578007 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:48.577975    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:48.590696 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:48.590671    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:49.040535 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:49.040505    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 19:59:55.231473 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.231440    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"]
Oct 30 19:59:55.236322 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.236299    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.291692 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.291659    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.291882 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.291726    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.291882 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.291752    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.291882 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.291806    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.292044 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.291976    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6mw\" (UniqueName: \"kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.292044 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.292025    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.342026 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.341994    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"premium-w5-407632-kserve-self-signed-certs\""
Oct 30 19:59:55.344253 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.344226    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"]
Oct 30 19:59:55.392846 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.392812    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6mw\" (UniqueName: \"kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393037 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.392924    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393037 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393023    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393160 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393085    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393160 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393109    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393160 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393139    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393473 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393450    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393579 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393545    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.393628 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.393592    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.395318 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.395294    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.395492 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.395475    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.455113 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.455077    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6mw\" (UniqueName: \"kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw\") pod \"premium-w5-407632-kserve-5d797b6cd5-fgfl9\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") " pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.547995 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.547911    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 19:59:55.940911 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:55.938212    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"]
Oct 30 19:59:56.067033 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:56.066994    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" event={"ID":"e02c50a5-c8a4-4382-a23c-a051e3dc23c4","Type":"ContainerStarted","Data":"51997c9cbb94742803c3f1f5b2c506f05615201554710b541e221b7e4e377a83"}
Oct 30 19:59:56.067217 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:56.067042    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" event={"ID":"e02c50a5-c8a4-4382-a23c-a051e3dc23c4","Type":"ContainerStarted","Data":"464db2870a771f51fdd6d46f0830c3d7cbb3bafb2431c8d89c1a71231d107723"}
Oct 30 19:59:56.187289 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:56.187175    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podStartSLOduration=1.187162802 podStartE2EDuration="1.187162802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:56.185620111 +0000 UTC m=+1092.350021751" watchObservedRunningTime="2026-10-30 19:59:56.187162802 +0000 UTC m=+1092.351564442"
Oct 30 19:59:56.548358 ip-10-0-133-27 kubenswrapper[4917]: I1030 19:59:56.548280    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:00:00.194501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.194467    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"]
Oct 30 20:00:00.197950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.197927    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.201326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.201306    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"]
Oct 30 20:00:00.205101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.205083    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"]
Oct 30 20:00:00.205207 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.205190    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.208870 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.208850    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.245255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.245226    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"]
Oct 30 20:00:00.253221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.251531    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:00.253221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.252276    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:00.254824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.254798    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"]
Oct 30 20:00:00.254926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.254882    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.258938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.258919    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.293840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.293815    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:00:00.297937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.296664    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"]
Oct 30 20:00:00.300971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.299501    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"]
Oct 30 20:00:00.305183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.305160    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"]
Oct 30 20:00:00.339630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339603    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpm9s\" (UniqueName: \"kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.339766 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339636    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.339766 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339671    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.339863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339796    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqgg\" (UniqueName: \"kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.339863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339825    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6d8k\" (UniqueName: \"kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.339921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339897    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjv6\" (UniqueName: \"kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.340006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.339984    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.340066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.340048    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ght9t\" (UniqueName: \"kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.340096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.340079    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.340179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.340164    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.345828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.345800    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"]
Oct 30 20:00:00.366897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.366868    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"]
Oct 30 20:00:00.441240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441201    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.441435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441283    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqgg\" (UniqueName: \"kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.441435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441326    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6d8k\" (UniqueName: \"kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.441435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441359    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjv6\" (UniqueName: \"kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.441435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441384    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.441435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441425    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ght9t\" (UniqueName: \"kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.441762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441448    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.441762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441493    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.441762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441528    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpm9s\" (UniqueName: \"kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.441762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.441556    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.442129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.442105    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.442312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.442286    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.442312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.442303    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.442437 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.442354    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.442488 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.442462    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.508764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.508697    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6d8k\" (UniqueName: \"kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.509391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.509371    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqgg\" (UniqueName: \"kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.512258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.512233    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:00.514896 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.514877    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ght9t\" (UniqueName: \"kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.517459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.517437    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpm9s\" (UniqueName: \"kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.517858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.517842    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjv6\" (UniqueName: \"kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.520670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.520652    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:00.527645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.527628    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:00.568512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.568186    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:00.577516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.577032    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:00.910886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.910857    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"]
Oct 30 20:00:00.912736 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:00.912704    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402f083b_8bdc_463a_8f2a_ed7ef6c1716e.slice/crio-630b45f27970c2cce84de2559e7fcfc418fbd9a56f88759fd0f76826806e5380 WatchSource:0}: Error finding container 630b45f27970c2cce84de2559e7fcfc418fbd9a56f88759fd0f76826806e5380: Status 404 returned error can't find the container with id 630b45f27970c2cce84de2559e7fcfc418fbd9a56f88759fd0f76826806e5380
Oct 30 20:00:00.957588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.957553    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"]
Oct 30 20:00:00.958346 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:00.958318    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c006e9_0466_4699_b360_491eb0f6a0dd.slice/crio-585de680df049971e1fe4cc4927296d70c1eb3ff3614757070e4a5a1aa5a652c WatchSource:0}: Error finding container 585de680df049971e1fe4cc4927296d70c1eb3ff3614757070e4a5a1aa5a652c: Status 404 returned error can't find the container with id 585de680df049971e1fe4cc4927296d70c1eb3ff3614757070e4a5a1aa5a652c
Oct 30 20:00:00.961263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:00.961198    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"]
Oct 30 20:00:00.961625 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:00.961598    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4102d7_4a84_4663_9ed3_cad3d7a85b5d.slice/crio-b5ebf05da7893f5a4e6dbea513cb87ed1aa339ecadd39d752c526b545b75f14f WatchSource:0}: Error finding container b5ebf05da7893f5a4e6dbea513cb87ed1aa339ecadd39d752c526b545b75f14f: Status 404 returned error can't find the container with id b5ebf05da7893f5a4e6dbea513cb87ed1aa339ecadd39d752c526b545b75f14f
Oct 30 20:00:01.012638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.012610    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"]
Oct 30 20:00:01.013794 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:01.013738    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae626af5_abf8_4d57_b841_cfe36e014fdc.slice/crio-bd1f44dd559c6140cebcc5c077533f5336cfe038dd1584c8be61ff2afaff97fe WatchSource:0}: Error finding container bd1f44dd559c6140cebcc5c077533f5336cfe038dd1584c8be61ff2afaff97fe: Status 404 returned error can't find the container with id bd1f44dd559c6140cebcc5c077533f5336cfe038dd1584c8be61ff2afaff97fe
Oct 30 20:00:01.016729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.016706    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"]
Oct 30 20:00:01.018421 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:01.018397    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf547edb2_f3b0_4ae0_b3a9_c8b1166ed32d.slice/crio-4c7417d1003851a4864a56e62810c329f60f098e255d9584093a9ae826a546b2 WatchSource:0}: Error finding container 4c7417d1003851a4864a56e62810c329f60f098e255d9584093a9ae826a546b2: Status 404 returned error can't find the container with id 4c7417d1003851a4864a56e62810c329f60f098e255d9584093a9ae826a546b2
Oct 30 20:00:01.093912 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.093879    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerStarted","Data":"4c7417d1003851a4864a56e62810c329f60f098e255d9584093a9ae826a546b2"}
Oct 30 20:00:01.095162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.095129    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" event={"ID":"af4102d7-4a84-4663-9ed3-cad3d7a85b5d","Type":"ContainerStarted","Data":"b5ebf05da7893f5a4e6dbea513cb87ed1aa339ecadd39d752c526b545b75f14f"}
Oct 30 20:00:01.096385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.096360    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" event={"ID":"39c006e9-0466-4699-b360-491eb0f6a0dd","Type":"ContainerStarted","Data":"585de680df049971e1fe4cc4927296d70c1eb3ff3614757070e4a5a1aa5a652c"}
Oct 30 20:00:01.097575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.097552    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" event={"ID":"ae626af5-abf8-4d57-b841-cfe36e014fdc","Type":"ContainerStarted","Data":"bd1f44dd559c6140cebcc5c077533f5336cfe038dd1584c8be61ff2afaff97fe"}
Oct 30 20:00:01.098691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:01.098669    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerStarted","Data":"630b45f27970c2cce84de2559e7fcfc418fbd9a56f88759fd0f76826806e5380"}
Oct 30 20:00:05.540213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:05.540177    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:00:05.553646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:05.548590    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:00:05.572756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:05.572484    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:00:06.166174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:06.166134    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:00:07.158218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.158182    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerStarted","Data":"633a289feb3049cb79c8f107a4812691b8e2b44fd9edbcbcf01a88fcbb0a0af1"}
Oct 30 20:00:07.159976 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.159953    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" event={"ID":"af4102d7-4a84-4663-9ed3-cad3d7a85b5d","Type":"ContainerStarted","Data":"41cdf532f16af87c0b5f5aa0addf0682fbb37dfb3a2bb00c6cf053e0041044e9"}
Oct 30 20:00:07.161631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.161604    4917 generic.go:356] "Generic (PLEG): container finished" podID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerID="9e8358e1566ec63f1bbdfa9cc9cd40a2d04c09c3b50c804276fff47be9e77963" exitCode=60
Oct 30 20:00:07.161733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.161684    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" event={"ID":"39c006e9-0466-4699-b360-491eb0f6a0dd","Type":"ContainerDied","Data":"9e8358e1566ec63f1bbdfa9cc9cd40a2d04c09c3b50c804276fff47be9e77963"}
Oct 30 20:00:07.161872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.161857    4917 scope.go:122] "RemoveContainer" containerID="9e8358e1566ec63f1bbdfa9cc9cd40a2d04c09c3b50c804276fff47be9e77963"
Oct 30 20:00:07.163308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.163282    4917 generic.go:356] "Generic (PLEG): container finished" podID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerID="f168eb1b82b3e91c19a49a8936bd3fddedd4be7f11fe78183f792b0ab59b265b" exitCode=60
Oct 30 20:00:07.163649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.163364    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" event={"ID":"ae626af5-abf8-4d57-b841-cfe36e014fdc","Type":"ContainerDied","Data":"f168eb1b82b3e91c19a49a8936bd3fddedd4be7f11fe78183f792b0ab59b265b"}
Oct 30 20:00:07.163649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.163530    4917 scope.go:122] "RemoveContainer" containerID="f168eb1b82b3e91c19a49a8936bd3fddedd4be7f11fe78183f792b0ab59b265b"
Oct 30 20:00:07.165387 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.165341    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerStarted","Data":"470b39299da019e2015fe89d2da9b54e32cee7705628adb3fb0e18ad6eee9389"}
Oct 30 20:00:07.276895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.276849    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" podStartSLOduration=1.886314797 podStartE2EDuration="7.276757086s" totalImagesPullingTime="5.390442289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.275165539 +0000 UTC m=+1103.439567198" watchObservedRunningTime="2026-10-30 20:00:07.276757086 +0000 UTC m=+1103.441158726"
Oct 30 20:00:07.654925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.654869    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" podStartSLOduration=2.162185083 podStartE2EDuration="7.654854154s" totalImagesPullingTime="5.492669071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.653760071 +0000 UTC m=+1103.818161713" watchObservedRunningTime="2026-10-30 20:00:07.654854154 +0000 UTC m=+1103.819255793"
Oct 30 20:00:07.789781 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:07.789725    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" podStartSLOduration=2.403023526 podStartE2EDuration="7.789712169s" totalImagesPullingTime="5.386688643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.787410868 +0000 UTC m=+1103.951812508" watchObservedRunningTime="2026-10-30 20:00:07.789712169 +0000 UTC m=+1103.954113819"
Oct 30 20:00:08.173237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.173201    4917 generic.go:356] "Generic (PLEG): container finished" podID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerID="b0d2da83cd6ccd06054209312f0c675fdf8132ec0a054eb2f2c69e69578e1fa2" exitCode=60
Oct 30 20:00:08.173674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.173288    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" event={"ID":"39c006e9-0466-4699-b360-491eb0f6a0dd","Type":"ContainerDied","Data":"b0d2da83cd6ccd06054209312f0c675fdf8132ec0a054eb2f2c69e69578e1fa2"}
Oct 30 20:00:08.173674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.173345    4917 scope.go:122] "RemoveContainer" containerID="9e8358e1566ec63f1bbdfa9cc9cd40a2d04c09c3b50c804276fff47be9e77963"
Oct 30 20:00:08.173674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.173597    4917 scope.go:122] "RemoveContainer" containerID="b0d2da83cd6ccd06054209312f0c675fdf8132ec0a054eb2f2c69e69578e1fa2"
Oct 30 20:00:08.173938 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:08.173915    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt_odh-ai-gateway-infra(39c006e9-0466-4699-b360-491eb0f6a0dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd"
Oct 30 20:00:08.176270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.176245    4917 generic.go:356] "Generic (PLEG): container finished" podID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerID="8377776301f084a6e2d6454c3db8259fcaff7a89257582caff13b7eed2c3f7c4" exitCode=60
Oct 30 20:00:08.176388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.176316    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" event={"ID":"ae626af5-abf8-4d57-b841-cfe36e014fdc","Type":"ContainerDied","Data":"8377776301f084a6e2d6454c3db8259fcaff7a89257582caff13b7eed2c3f7c4"}
Oct 30 20:00:08.176581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.176568    4917 scope.go:122] "RemoveContainer" containerID="8377776301f084a6e2d6454c3db8259fcaff7a89257582caff13b7eed2c3f7c4"
Oct 30 20:00:08.176841 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:08.176823    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx_odh-ai-gateway-infra(ae626af5-abf8-4d57-b841-cfe36e014fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc"
Oct 30 20:00:08.178050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.178029    4917 generic.go:356] "Generic (PLEG): container finished" podID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerID="470b39299da019e2015fe89d2da9b54e32cee7705628adb3fb0e18ad6eee9389" exitCode=60
Oct 30 20:00:08.178143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.178118    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerDied","Data":"470b39299da019e2015fe89d2da9b54e32cee7705628adb3fb0e18ad6eee9389"}
Oct 30 20:00:08.178313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.178298    4917 scope.go:122] "RemoveContainer" containerID="470b39299da019e2015fe89d2da9b54e32cee7705628adb3fb0e18ad6eee9389"
Oct 30 20:00:08.179578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.179556    4917 generic.go:356] "Generic (PLEG): container finished" podID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerID="633a289feb3049cb79c8f107a4812691b8e2b44fd9edbcbcf01a88fcbb0a0af1" exitCode=60
Oct 30 20:00:08.179674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.179627    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerDied","Data":"633a289feb3049cb79c8f107a4812691b8e2b44fd9edbcbcf01a88fcbb0a0af1"}
Oct 30 20:00:08.180066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.179938    4917 scope.go:122] "RemoveContainer" containerID="633a289feb3049cb79c8f107a4812691b8e2b44fd9edbcbcf01a88fcbb0a0af1"
Oct 30 20:00:08.181528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.181505    4917 generic.go:356] "Generic (PLEG): container finished" podID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerID="41cdf532f16af87c0b5f5aa0addf0682fbb37dfb3a2bb00c6cf053e0041044e9" exitCode=60
Oct 30 20:00:08.181633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.181557    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" event={"ID":"af4102d7-4a84-4663-9ed3-cad3d7a85b5d","Type":"ContainerDied","Data":"41cdf532f16af87c0b5f5aa0addf0682fbb37dfb3a2bb00c6cf053e0041044e9"}
Oct 30 20:00:08.181846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.181765    4917 scope.go:122] "RemoveContainer" containerID="41cdf532f16af87c0b5f5aa0addf0682fbb37dfb3a2bb00c6cf053e0041044e9"
Oct 30 20:00:08.187244 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:08.187217    4917 scope.go:122] "RemoveContainer" containerID="f168eb1b82b3e91c19a49a8936bd3fddedd4be7f11fe78183f792b0ab59b265b"
Oct 30 20:00:09.188515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.188493    4917 scope.go:122] "RemoveContainer" containerID="b0d2da83cd6ccd06054209312f0c675fdf8132ec0a054eb2f2c69e69578e1fa2"
Oct 30 20:00:09.188905 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:09.188746    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt_odh-ai-gateway-infra(39c006e9-0466-4699-b360-491eb0f6a0dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd"
Oct 30 20:00:09.190009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.189993    4917 scope.go:122] "RemoveContainer" containerID="8377776301f084a6e2d6454c3db8259fcaff7a89257582caff13b7eed2c3f7c4"
Oct 30 20:00:09.190227 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:09.190203    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx_odh-ai-gateway-infra(ae626af5-abf8-4d57-b841-cfe36e014fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc"
Oct 30 20:00:09.191628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.191608    4917 generic.go:356] "Generic (PLEG): container finished" podID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerID="515056ed6902bafe6ffe8e2c10c4ab7950277415faf6d1055397cfdba7be94f7" exitCode=60
Oct 30 20:00:09.191705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.191684    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerDied","Data":"515056ed6902bafe6ffe8e2c10c4ab7950277415faf6d1055397cfdba7be94f7"}
Oct 30 20:00:09.191753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.191728    4917 scope.go:122] "RemoveContainer" containerID="470b39299da019e2015fe89d2da9b54e32cee7705628adb3fb0e18ad6eee9389"
Oct 30 20:00:09.191867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.191852    4917 scope.go:122] "RemoveContainer" containerID="515056ed6902bafe6ffe8e2c10c4ab7950277415faf6d1055397cfdba7be94f7"
Oct 30 20:00:09.192051 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:09.192033    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n_odh-ai-gateway-infra(402f083b-8bdc-463a-8f2a-ed7ef6c1716e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e"
Oct 30 20:00:09.193501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.193483    4917 generic.go:356] "Generic (PLEG): container finished" podID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerID="fcbe6d0cd81236a18a04c3659d39c52c372898933374ea76720fc4d2ebacf854" exitCode=60
Oct 30 20:00:09.193591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.193554    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerDied","Data":"fcbe6d0cd81236a18a04c3659d39c52c372898933374ea76720fc4d2ebacf854"}
Oct 30 20:00:09.193712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.193698    4917 scope.go:122] "RemoveContainer" containerID="fcbe6d0cd81236a18a04c3659d39c52c372898933374ea76720fc4d2ebacf854"
Oct 30 20:00:09.193887 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:09.193861    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47_odh-ai-gateway-infra(f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d"
Oct 30 20:00:09.195211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.195191    4917 generic.go:356] "Generic (PLEG): container finished" podID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerID="a7f5d540e905aab167de2502015b0f42c3a3d192f920b1c0529fef4baa212c8f" exitCode=60
Oct 30 20:00:09.195295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.195232    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" event={"ID":"af4102d7-4a84-4663-9ed3-cad3d7a85b5d","Type":"ContainerDied","Data":"a7f5d540e905aab167de2502015b0f42c3a3d192f920b1c0529fef4baa212c8f"}
Oct 30 20:00:09.195446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.195431    4917 scope.go:122] "RemoveContainer" containerID="a7f5d540e905aab167de2502015b0f42c3a3d192f920b1c0529fef4baa212c8f"
Oct 30 20:00:09.195611 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:09.195593    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm_odh-ai-gateway-infra(af4102d7-4a84-4663-9ed3-cad3d7a85b5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d"
Oct 30 20:00:09.202735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.202719    4917 scope.go:122] "RemoveContainer" containerID="633a289feb3049cb79c8f107a4812691b8e2b44fd9edbcbcf01a88fcbb0a0af1"
Oct 30 20:00:09.214120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:09.214103    4917 scope.go:122] "RemoveContainer" containerID="41cdf532f16af87c0b5f5aa0addf0682fbb37dfb3a2bb00c6cf053e0041044e9"
Oct 30 20:00:09.679801 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:09.679744    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:00:10.202600 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:10.202563    4917 scope.go:122] "RemoveContainer" containerID="515056ed6902bafe6ffe8e2c10c4ab7950277415faf6d1055397cfdba7be94f7"
Oct 30 20:00:10.203050 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:10.202832    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n_odh-ai-gateway-infra(402f083b-8bdc-463a-8f2a-ed7ef6c1716e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e"
Oct 30 20:00:10.204088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:10.204067    4917 scope.go:122] "RemoveContainer" containerID="fcbe6d0cd81236a18a04c3659d39c52c372898933374ea76720fc4d2ebacf854"
Oct 30 20:00:10.204318 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:10.204294    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47_odh-ai-gateway-infra(f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d"
Oct 30 20:00:10.205502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:10.205489    4917 scope.go:122] "RemoveContainer" containerID="a7f5d540e905aab167de2502015b0f42c3a3d192f920b1c0529fef4baa212c8f"
Oct 30 20:00:10.205623 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:10.205609    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm_odh-ai-gateway-infra(af4102d7-4a84-4663-9ed3-cad3d7a85b5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d"
Oct 30 20:00:18.995365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:18.995330    4917 scope.go:122] "RemoveContainer" containerID="8377776301f084a6e2d6454c3db8259fcaff7a89257582caff13b7eed2c3f7c4"
Oct 30 20:00:19.263732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:19.263700    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" event={"ID":"ae626af5-abf8-4d57-b841-cfe36e014fdc","Type":"ContainerStarted","Data":"394026040b2d2ff05667ff70d3cfd4baa1eb91befcc5c7fa3b332aa4dba033b2"}
Oct 30 20:00:19.382501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:19.382446    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" podStartSLOduration=14.270162478 podStartE2EDuration="19.38243389s" totalImagesPullingTime="5.112271412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:19.381274469 +0000 UTC m=+1115.545676108" watchObservedRunningTime="2026-10-30 20:00:19.38243389 +0000 UTC m=+1115.546835529"
Oct 30 20:00:20.271060 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:20.271025    4917 generic.go:356] "Generic (PLEG): container finished" podID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerID="394026040b2d2ff05667ff70d3cfd4baa1eb91befcc5c7fa3b332aa4dba033b2" exitCode=60
Oct 30 20:00:20.271506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:20.271096    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" event={"ID":"ae626af5-abf8-4d57-b841-cfe36e014fdc","Type":"ContainerDied","Data":"394026040b2d2ff05667ff70d3cfd4baa1eb91befcc5c7fa3b332aa4dba033b2"}
Oct 30 20:00:20.271506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:20.271160    4917 scope.go:122] "RemoveContainer" containerID="8377776301f084a6e2d6454c3db8259fcaff7a89257582caff13b7eed2c3f7c4"
Oct 30 20:00:20.271506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:20.271505    4917 scope.go:122] "RemoveContainer" containerID="394026040b2d2ff05667ff70d3cfd4baa1eb91befcc5c7fa3b332aa4dba033b2"
Oct 30 20:00:20.271836 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:20.271812    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx_odh-ai-gateway-infra(ae626af5-abf8-4d57-b841-cfe36e014fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc"
Oct 30 20:00:20.401206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:20.401177    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"]
Oct 30 20:00:20.995043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:20.995013    4917 scope.go:122] "RemoveContainer" containerID="515056ed6902bafe6ffe8e2c10c4ab7950277415faf6d1055397cfdba7be94f7"
Oct 30 20:00:21.278969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.278940    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerStarted","Data":"4b487053957da42b456b26685fa9cc95c6eb5971ef3027495b72d4aaccfaf8fc"}
Oct 30 20:00:21.419858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.419836    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:21.555330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.555245    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca\") pod \"ae626af5-abf8-4d57-b841-cfe36e014fdc\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") "
Oct 30 20:00:21.555495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.555340    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t\" (UniqueName: \"kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t\") pod \"ae626af5-abf8-4d57-b841-cfe36e014fdc\" (UID: \"ae626af5-abf8-4d57-b841-cfe36e014fdc\") "
Oct 30 20:00:21.555638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.555613    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca" pod "ae626af5-abf8-4d57-b841-cfe36e014fdc" (UID: "ae626af5-abf8-4d57-b841-cfe36e014fdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.557402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.557376    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t" pod "ae626af5-abf8-4d57-b841-cfe36e014fdc" (UID: "ae626af5-abf8-4d57-b841-cfe36e014fdc"). InnerVolumeSpecName "kube-api-access-ght9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.656245 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.656207    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae626af5-abf8-4d57-b841-cfe36e014fdc-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.656245 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.656242    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ght9t\" (UniqueName: \"kubernetes.io/projected/ae626af5-abf8-4d57-b841-cfe36e014fdc-kube-api-access-ght9t\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.995823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.995791    4917 scope.go:122] "RemoveContainer" containerID="b0d2da83cd6ccd06054209312f0c675fdf8132ec0a054eb2f2c69e69578e1fa2"
Oct 30 20:00:21.996004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:21.995942    4917 scope.go:122] "RemoveContainer" containerID="a7f5d540e905aab167de2502015b0f42c3a3d192f920b1c0529fef4baa212c8f"
Oct 30 20:00:22.288073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.288043    4917 generic.go:356] "Generic (PLEG): container finished" podID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerID="4b487053957da42b456b26685fa9cc95c6eb5971ef3027495b72d4aaccfaf8fc" exitCode=60
Oct 30 20:00:22.288495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.288116    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerDied","Data":"4b487053957da42b456b26685fa9cc95c6eb5971ef3027495b72d4aaccfaf8fc"}
Oct 30 20:00:22.288495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.288191    4917 scope.go:122] "RemoveContainer" containerID="515056ed6902bafe6ffe8e2c10c4ab7950277415faf6d1055397cfdba7be94f7"
Oct 30 20:00:22.288621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.288521    4917 scope.go:122] "RemoveContainer" containerID="4b487053957da42b456b26685fa9cc95c6eb5971ef3027495b72d4aaccfaf8fc"
Oct 30 20:00:22.288810 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:22.288779    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n_odh-ai-gateway-infra(402f083b-8bdc-463a-8f2a-ed7ef6c1716e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e"
Oct 30 20:00:22.289556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.289537    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx" event={"ID":"ae626af5-abf8-4d57-b841-cfe36e014fdc","Type":"ContainerDied","Data":"bd1f44dd559c6140cebcc5c077533f5336cfe038dd1584c8be61ff2afaff97fe"}
Oct 30 20:00:22.289625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.289556    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"
Oct 30 20:00:22.300739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.300716    4917 scope.go:122] "RemoveContainer" containerID="394026040b2d2ff05667ff70d3cfd4baa1eb91befcc5c7fa3b332aa4dba033b2"
Oct 30 20:00:22.416732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.416702    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"]
Oct 30 20:00:22.519219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.519141    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"]
Oct 30 20:00:22.528595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.528568    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889840-nvgkx"]
Oct 30 20:00:22.616138 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.616110    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:00:22.995301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:22.995272    4917 scope.go:122] "RemoveContainer" containerID="fcbe6d0cd81236a18a04c3659d39c52c372898933374ea76720fc4d2ebacf854"
Oct 30 20:00:23.298150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.298121    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerStarted","Data":"1043a78ae5775a57c411146ac62b921150f5a3a7d3734b232921c07a44e2a614"}
Oct 30 20:00:23.299992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.299973    4917 generic.go:356] "Generic (PLEG): container finished" podID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerID="c64694734f541c153045ed19060cf8572062d94605608ec79df54d5d7a579968" exitCode=60
Oct 30 20:00:23.300101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.300045    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" event={"ID":"af4102d7-4a84-4663-9ed3-cad3d7a85b5d","Type":"ContainerDied","Data":"c64694734f541c153045ed19060cf8572062d94605608ec79df54d5d7a579968"}
Oct 30 20:00:23.300142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.300104    4917 scope.go:122] "RemoveContainer" containerID="a7f5d540e905aab167de2502015b0f42c3a3d192f920b1c0529fef4baa212c8f"
Oct 30 20:00:23.300429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.300408    4917 scope.go:122] "RemoveContainer" containerID="c64694734f541c153045ed19060cf8572062d94605608ec79df54d5d7a579968"
Oct 30 20:00:23.300675 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:23.300656    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm_odh-ai-gateway-infra(af4102d7-4a84-4663-9ed3-cad3d7a85b5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d"
Oct 30 20:00:23.301956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.301937    4917 generic.go:356] "Generic (PLEG): container finished" podID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerID="f4c4bfb1818c58418b24358a639eadd2aedddee364d11e8ee4b2da4ab7e7be15" exitCode=60
Oct 30 20:00:23.302058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.302012    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" event={"ID":"39c006e9-0466-4699-b360-491eb0f6a0dd","Type":"ContainerDied","Data":"f4c4bfb1818c58418b24358a639eadd2aedddee364d11e8ee4b2da4ab7e7be15"}
Oct 30 20:00:23.302330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.302316    4917 scope.go:122] "RemoveContainer" containerID="f4c4bfb1818c58418b24358a639eadd2aedddee364d11e8ee4b2da4ab7e7be15"
Oct 30 20:00:23.302538 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:23.302517    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt_odh-ai-gateway-infra(39c006e9-0466-4699-b360-491eb0f6a0dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd"
Oct 30 20:00:23.312320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.312300    4917 scope.go:122] "RemoveContainer" containerID="b0d2da83cd6ccd06054209312f0c675fdf8132ec0a054eb2f2c69e69578e1fa2"
Oct 30 20:00:23.444019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.443988    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:23.573334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.573257    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg\" (UniqueName: \"kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg\") pod \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") "
Oct 30 20:00:23.573479 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.573345    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca\") pod \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\" (UID: \"402f083b-8bdc-463a-8f2a-ed7ef6c1716e\") "
Oct 30 20:00:23.573666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.573646    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca" pod "402f083b-8bdc-463a-8f2a-ed7ef6c1716e" (UID: "402f083b-8bdc-463a-8f2a-ed7ef6c1716e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.575310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.575279    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg" pod "402f083b-8bdc-463a-8f2a-ed7ef6c1716e" (UID: "402f083b-8bdc-463a-8f2a-ed7ef6c1716e"). InnerVolumeSpecName "kube-api-access-2rqgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.674745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.674692    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rqgg\" (UniqueName: \"kubernetes.io/projected/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-kube-api-access-2rqgg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:23.674745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:23.674715    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f083b-8bdc-463a-8f2a-ed7ef6c1716e-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.001679 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.001644    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" path="/var/lib/kubelet/pods/ae626af5-abf8-4d57-b841-cfe36e014fdc/volumes"
Oct 30 20:00:24.310420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.310387    4917 generic.go:356] "Generic (PLEG): container finished" podID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerID="1043a78ae5775a57c411146ac62b921150f5a3a7d3734b232921c07a44e2a614" exitCode=60
Oct 30 20:00:24.310895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.310458    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerDied","Data":"1043a78ae5775a57c411146ac62b921150f5a3a7d3734b232921c07a44e2a614"}
Oct 30 20:00:24.310895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.310549    4917 scope.go:122] "RemoveContainer" containerID="fcbe6d0cd81236a18a04c3659d39c52c372898933374ea76720fc4d2ebacf854"
Oct 30 20:00:24.310982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.310907    4917 scope.go:122] "RemoveContainer" containerID="1043a78ae5775a57c411146ac62b921150f5a3a7d3734b232921c07a44e2a614"
Oct 30 20:00:24.311187 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:24.311160    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47_odh-ai-gateway-infra(f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d"
Oct 30 20:00:24.314752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.314733    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n" event={"ID":"402f083b-8bdc-463a-8f2a-ed7ef6c1716e","Type":"ContainerDied","Data":"630b45f27970c2cce84de2559e7fcfc418fbd9a56f88759fd0f76826806e5380"}
Oct 30 20:00:24.314858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.314797    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"
Oct 30 20:00:24.322265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.322247    4917 scope.go:122] "RemoveContainer" containerID="4b487053957da42b456b26685fa9cc95c6eb5971ef3027495b72d4aaccfaf8fc"
Oct 30 20:00:24.544088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.544058    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"]
Oct 30 20:00:24.548454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.548426    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889840-jws4n"]
Oct 30 20:00:24.563496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.563435    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"]
Oct 30 20:00:24.690688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.690659    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"]
Oct 30 20:00:24.701879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.701852    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:24.782549 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.782519    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca\") pod \"39c006e9-0466-4699-b360-491eb0f6a0dd\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") "
Oct 30 20:00:24.782696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.782584    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k\" (UniqueName: \"kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k\") pod \"39c006e9-0466-4699-b360-491eb0f6a0dd\" (UID: \"39c006e9-0466-4699-b360-491eb0f6a0dd\") "
Oct 30 20:00:24.782886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.782864    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca" pod "39c006e9-0466-4699-b360-491eb0f6a0dd" (UID: "39c006e9-0466-4699-b360-491eb0f6a0dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:24.784555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.784529    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k" pod "39c006e9-0466-4699-b360-491eb0f6a0dd" (UID: "39c006e9-0466-4699-b360-491eb0f6a0dd"). InnerVolumeSpecName "kube-api-access-t6d8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:24.804323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.804299    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"]
Oct 30 20:00:24.883357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.883322    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6d8k\" (UniqueName: \"kubernetes.io/projected/39c006e9-0466-4699-b360-491eb0f6a0dd-kube-api-access-t6d8k\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.883357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.883358    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c006e9-0466-4699-b360-491eb0f6a0dd-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.944309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.944287    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:24.983712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.983685    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca\") pod \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") "
Oct 30 20:00:24.983867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.983730    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6\" (UniqueName: \"kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6\") pod \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\" (UID: \"af4102d7-4a84-4663-9ed3-cad3d7a85b5d\") "
Oct 30 20:00:24.984083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.984060    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca" pod "af4102d7-4a84-4663-9ed3-cad3d7a85b5d" (UID: "af4102d7-4a84-4663-9ed3-cad3d7a85b5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:24.985715 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:24.985696    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6" pod "af4102d7-4a84-4663-9ed3-cad3d7a85b5d" (UID: "af4102d7-4a84-4663-9ed3-cad3d7a85b5d"). InnerVolumeSpecName "kube-api-access-xdjv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:25.084374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.084344    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:25.084374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.084368    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjv6\" (UniqueName: \"kubernetes.io/projected/af4102d7-4a84-4663-9ed3-cad3d7a85b5d-kube-api-access-xdjv6\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:25.324839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.324815    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"
Oct 30 20:00:25.325190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.324866    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm" event={"ID":"af4102d7-4a84-4663-9ed3-cad3d7a85b5d","Type":"ContainerDied","Data":"b5ebf05da7893f5a4e6dbea513cb87ed1aa339ecadd39d752c526b545b75f14f"}
Oct 30 20:00:25.325190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.324922    4917 scope.go:122] "RemoveContainer" containerID="c64694734f541c153045ed19060cf8572062d94605608ec79df54d5d7a579968"
Oct 30 20:00:25.326345 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.326315    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"
Oct 30 20:00:25.326459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.326318    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt" event={"ID":"39c006e9-0466-4699-b360-491eb0f6a0dd","Type":"ContainerDied","Data":"585de680df049971e1fe4cc4927296d70c1eb3ff3614757070e4a5a1aa5a652c"}
Oct 30 20:00:25.337117 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.337097    4917 scope.go:122] "RemoveContainer" containerID="f4c4bfb1818c58418b24358a639eadd2aedddee364d11e8ee4b2da4ab7e7be15"
Oct 30 20:00:25.452691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.452666    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:25.486675 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.486652    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s\" (UniqueName: \"kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s\") pod \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") "
Oct 30 20:00:25.486855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.486727    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca\") pod \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\" (UID: \"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d\") "
Oct 30 20:00:25.487097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.487077    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca" pod "f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" (UID: "f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:25.488649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.488630    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s" pod "f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" (UID: "f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d"). InnerVolumeSpecName "kube-api-access-wpm9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:25.561102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.561072    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"]
Oct 30 20:00:25.565764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.565739    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889840-jrnzt"]
Oct 30 20:00:25.587458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.587403    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:25.587458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.587426    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpm9s\" (UniqueName: \"kubernetes.io/projected/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d-kube-api-access-wpm9s\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:25.684065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.684033    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"]
Oct 30 20:00:25.687454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:25.687422    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889840-4pqcm"]
Oct 30 20:00:26.002119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.002093    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" path="/var/lib/kubelet/pods/39c006e9-0466-4699-b360-491eb0f6a0dd/volumes"
Oct 30 20:00:26.002443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.002430    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" path="/var/lib/kubelet/pods/402f083b-8bdc-463a-8f2a-ed7ef6c1716e/volumes"
Oct 30 20:00:26.002761 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.002749    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" path="/var/lib/kubelet/pods/af4102d7-4a84-4663-9ed3-cad3d7a85b5d/volumes"
Oct 30 20:00:26.340396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.340296    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47" event={"ID":"f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d","Type":"ContainerDied","Data":"4c7417d1003851a4864a56e62810c329f60f098e255d9584093a9ae826a546b2"}
Oct 30 20:00:26.340396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.340342    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"
Oct 30 20:00:26.340978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.340404    4917 scope.go:122] "RemoveContainer" containerID="1043a78ae5775a57c411146ac62b921150f5a3a7d3734b232921c07a44e2a614"
Oct 30 20:00:26.507616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.507582    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"]
Oct 30 20:00:26.509663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:26.509642    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889840-7pp47"]
Oct 30 20:00:28.002069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.002037    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" path="/var/lib/kubelet/pods/f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d/volumes"
Oct 30 20:00:28.012084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012047    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"]
Oct 30 20:00:28.012506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012488    4917 cpu_manager.go:414] "containerMap: removing container" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012507    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012532    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012541    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012555    4917 cpu_manager.go:414] "containerMap: removing container" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012562    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012572    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012579    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012590    4917 cpu_manager.go:414] "containerMap: removing container" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:28.012597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012597    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012615    4917 cpu_manager.go:414] "containerMap: removing container" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012622    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012636    4917 cpu_manager.go:414] "containerMap: removing container" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012643    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012652    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012659    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012683    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012690    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012710    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012717    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012727    4917 cpu_manager.go:414] "containerMap: removing container" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012733    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012870    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012882    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012890    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012900    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012908    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012920    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012929    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012936    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012945    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012956    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:28.013033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.012967    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:28.017798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.017759    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.072928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.072898    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-407632-openshift-default-dockercfg-5j7qg\""
Oct 30 20:00:28.105113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105084    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrbh\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105157    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105259    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105298    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105325    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105394    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105430    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105452    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.105566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.105534    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.125464 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.125438    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"]
Oct 30 20:00:28.206099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206064    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206105    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206125    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206153    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206184    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrbh\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206275    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206317    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206351    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206376    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206525    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206696    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206818    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.206883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.206850    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.207076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.207058    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.208669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.208649    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.208820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.208802    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.269569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.269507    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.269685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.269596    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrbh\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.330399 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.330368    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:28.764740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.764717    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"]
Oct 30 20:00:28.766007 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:28.765977    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fd940b_caf5_410b_93e5_45759775b5bf.slice/crio-1514831ddb8a82966090023e90c0a6f40eb7442220c948d0083772d1d0949f7c WatchSource:0}: Error finding container 1514831ddb8a82966090023e90c0a6f40eb7442220c948d0083772d1d0949f7c: Status 404 returned error can't find the container with id 1514831ddb8a82966090023e90c0a6f40eb7442220c948d0083772d1d0949f7c
Oct 30 20:00:28.768261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.768234    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:28.768323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.768292    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:28.768356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:28.768324    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:00:29.372344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:29.372307    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" event={"ID":"e3fd940b-caf5-410b-93e5-45759775b5bf","Type":"ContainerStarted","Data":"de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85"}
Oct 30 20:00:29.372726 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:29.372359    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" event={"ID":"e3fd940b-caf5-410b-93e5-45759775b5bf","Type":"ContainerStarted","Data":"1514831ddb8a82966090023e90c0a6f40eb7442220c948d0083772d1d0949f7c"}
Oct 30 20:00:29.503961 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:29.503900    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podStartSLOduration=2.503878172 podStartE2EDuration="2.503878172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:29.50285831 +0000 UTC m=+1125.667259949" watchObservedRunningTime="2026-10-30 20:00:29.503878172 +0000 UTC m=+1125.668279814"
Oct 30 20:00:30.331001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:30.330971    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:30.332090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:30.332062    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:00:30.332201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:30.332108    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 20:00:31.331288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:31.331251    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:00:31.331667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:31.331308    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 20:00:32.331325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.331287    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:00:32.331742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.331355    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 20:00:32.619658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.619566    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"]
Oct 30 20:00:32.620012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.619995    4917 cpu_manager.go:414] "containerMap: removing container" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:32.620012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620012    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620028    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620033    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae626af5-abf8-4d57-b841-cfe36e014fdc" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620047    4917 cpu_manager.go:414] "containerMap: removing container" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620053    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620061    4917 cpu_manager.go:414] "containerMap: removing container" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620065    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:32.620142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620139    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="402f083b-8bdc-463a-8f2a-ed7ef6c1716e" containerName="cleanup"
Oct 30 20:00:32.620472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620152    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f547edb2-f3b0-4ae0-b3a9-c8b1166ed32d" containerName="cleanup"
Oct 30 20:00:32.620472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620158    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c006e9-0466-4699-b360-491eb0f6a0dd" containerName="cleanup"
Oct 30 20:00:32.620472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.620165    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4102d7-4a84-4663-9ed3-cad3d7a85b5d" containerName="cleanup"
Oct 30 20:00:32.623324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.623303    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.668543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.668509    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.668920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.668894    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.669143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.669117    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.669263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.669155    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.669263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.669208    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.669263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.669246    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chd2j\" (UniqueName: \"kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.682897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.682875    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"]
Oct 30 20:00:32.729691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.729663    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"premium-w1-3ccc9f-kserve-self-signed-certs\""
Oct 30 20:00:32.770183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770152    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.770391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770218    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.770391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770255    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chd2j\" (UniqueName: \"kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.770391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770319    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.770391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770360    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.770652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770431    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.770652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.770599    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.771204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.771179    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.771420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.771355    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.773562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.773534    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.773804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.773785    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.834783 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.834749    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chd2j\" (UniqueName: \"kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j\") pod \"premium-w1-3ccc9f-kserve-658f4b8656-rlhqp\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") " pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:32.935669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:32.935630    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:33.331395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:33.331316    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:00:33.331395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:33.331369    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 20:00:33.366796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:33.366439    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"]
Oct 30 20:00:33.370856 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:33.370818    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b5de93_7a58_48c2_ae9a_63483ec8265b.slice/crio-aa9daaf67e3e6f462bf0038c80b1c8fb8e2f5c0932a0750a116ee725cf02a99f WatchSource:0}: Error finding container aa9daaf67e3e6f462bf0038c80b1c8fb8e2f5c0932a0750a116ee725cf02a99f: Status 404 returned error can't find the container with id aa9daaf67e3e6f462bf0038c80b1c8fb8e2f5c0932a0750a116ee725cf02a99f
Oct 30 20:00:33.395197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:33.395176    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" event={"ID":"99b5de93-7a58-48c2-ae9a-63483ec8265b","Type":"ContainerStarted","Data":"aa9daaf67e3e6f462bf0038c80b1c8fb8e2f5c0932a0750a116ee725cf02a99f"}
Oct 30 20:00:34.331392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:34.331354    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:00:34.331761 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:34.331407    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 20:00:34.401756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:34.401719    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" event={"ID":"99b5de93-7a58-48c2-ae9a-63483ec8265b","Type":"ContainerStarted","Data":"6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53"}
Oct 30 20:00:34.525224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:34.525173    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podStartSLOduration=2.525159648 podStartE2EDuration="2.525159648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:34.521669133 +0000 UTC m=+1130.686070774" watchObservedRunningTime="2026-10-30 20:00:34.525159648 +0000 UTC m=+1130.689561289"
Oct 30 20:00:34.936046 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:34.936017    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:35.337279 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:35.337199    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:35.337737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:35.337563    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:35.338888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:35.338863    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:00:39.677913 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:00:39.677879    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:00:42.936720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:42.936691    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:42.949237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:42.949213    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:43.461002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:43.460971    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:00:54.180658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:54.180621    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:00:58.261370 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.261340    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"]
Oct 30 20:00:58.268649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.268627    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.324952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.324920    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"]
Oct 30 20:00:58.372414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.372389    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"kube-root-ca.crt\""
Oct 30 20:00:58.372639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.372621    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"premium-w3-f84d85-kserve-self-signed-certs\""
Oct 30 20:00:58.372721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.372666    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"default-dockercfg-8kngq\""
Oct 30 20:00:58.372833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.372814    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"openshift-service-ca.crt\""
Oct 30 20:00:58.426711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.426679    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdnk\" (UniqueName: \"kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.426874 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.426724    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.426874 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.426838    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.426967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.426874    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.426967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.426910    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.426967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.426929    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.527817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.527699    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.527817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.527752    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.527859    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdnk\" (UniqueName: \"kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.527910    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.527964    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.528000    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.528164    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.528246    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.528468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.528448    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.530834 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.530812    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.531305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.531287    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.590461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.590434    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdnk\" (UniqueName: \"kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk\") pod \"premium-w3-f84d85-kserve-64447cd98d-ffzcl\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") " pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:58.880588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:58.880505    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:00:59.256513 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:59.256490    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"]
Oct 30 20:00:59.258274 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:00:59.258242    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51341bce_04de_434a_aea0_a884a76dd5c8.slice/crio-3984fd1c0a05322ba7e107f367f9ee88cf17c1bb1bb5d28248da42ddd291fa87 WatchSource:0}: Error finding container 3984fd1c0a05322ba7e107f367f9ee88cf17c1bb1bb5d28248da42ddd291fa87: Status 404 returned error can't find the container with id 3984fd1c0a05322ba7e107f367f9ee88cf17c1bb1bb5d28248da42ddd291fa87
Oct 30 20:00:59.541467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:59.541360    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" event={"ID":"51341bce-04de-434a-aea0-a884a76dd5c8","Type":"ContainerStarted","Data":"f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5"}
Oct 30 20:00:59.541467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:59.541409    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" event={"ID":"51341bce-04de-434a-aea0-a884a76dd5c8","Type":"ContainerStarted","Data":"3984fd1c0a05322ba7e107f367f9ee88cf17c1bb1bb5d28248da42ddd291fa87"}
Oct 30 20:00:59.666893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:59.666832    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podStartSLOduration=1.666809512 podStartE2EDuration="1.666809512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:59.663889825 +0000 UTC m=+1155.828291467" watchObservedRunningTime="2026-10-30 20:00:59.666809512 +0000 UTC m=+1155.831211153"
Oct 30 20:00:59.881713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:00:59.881630    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:01:07.509276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.509235    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"]
Oct 30 20:01:07.513625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.513602    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.568955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.568928    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:07.612562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.612524    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzkc\" (UniqueName: \"kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.612732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.612691    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.612732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.612724    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.619779 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.619745    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:07.619895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.619783    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:07.620016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.620000    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:07.622381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.622357    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"]
Oct 30 20:01:07.713508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.713461    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.713679 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.713609    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzkc\" (UniqueName: \"kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.713736 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.713717    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.714357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.714332    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.715875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.715851    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.776179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.776111    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzkc\" (UniqueName: \"kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc\") pod \"maas-api-revoke-sub-e8042c72-95h6r\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:07.829246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:07.829214    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:08.218973 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.218943    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"]
Oct 30 20:01:08.219197 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:08.219165    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a955af_47dd_4248_b353_9b3ba02e9f2f.slice/crio-a4a1faedfcfe1f5c062ba15d1854e3f0a61311d843f28862dc80bae378c38b71 WatchSource:0}: Error finding container a4a1faedfcfe1f5c062ba15d1854e3f0a61311d843f28862dc80bae378c38b71: Status 404 returned error can't find the container with id a4a1faedfcfe1f5c062ba15d1854e3f0a61311d843f28862dc80bae378c38b71
Oct 30 20:01:08.421814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.421781    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"]
Oct 30 20:01:08.426687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.426664    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.483254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.483064    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"premium-w2-d67ea9-kserve-self-signed-certs\""
Oct 30 20:01:08.486158 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.486136    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"]
Oct 30 20:01:08.605600 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.605564    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r" event={"ID":"c8a955af-47dd-4248-b353-9b3ba02e9f2f","Type":"ContainerStarted","Data":"df7af9bb40ddedda2e6a0f6d1d9e117796c2609af02fb8d7876b466320ee6510"}
Oct 30 20:01:08.606048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.605647    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r" event={"ID":"c8a955af-47dd-4248-b353-9b3ba02e9f2f","Type":"ContainerStarted","Data":"a4a1faedfcfe1f5c062ba15d1854e3f0a61311d843f28862dc80bae378c38b71"}
Oct 30 20:01:08.622212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.622180    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.622359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.622238    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.622402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.622370    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgtp\" (UniqueName: \"kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.622510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.622485    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.622558 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.622543    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.622589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.622576    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.723603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.723555    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.723761 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.723658    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgtp\" (UniqueName: \"kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.724123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.724099    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.724123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.724114    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.724304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.724172    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.724304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.724200    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.724304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.724290    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.724464 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.724398    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.725115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.725082    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.726332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.726289    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r" podStartSLOduration=1.726274343 podStartE2EDuration="1.726274343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:08.725036293 +0000 UTC m=+1164.889437936" watchObservedRunningTime="2026-10-30 20:01:08.726274343 +0000 UTC m=+1164.890675984"
Oct 30 20:01:08.727201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.727170    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.727693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.727669    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.786276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.786192    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgtp\" (UniqueName: \"kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp\") pod \"premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") " pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:08.881277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.881246    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:01:08.894285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:08.894261    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:01:09.038900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.038819    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:09.420490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.420464    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"]
Oct 30 20:01:09.421840 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:09.421797    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda88a932e_0217_4f8f_a956_54e5cef8e7b1.slice/crio-478b55165f7266edb7fcc1290861751049be43ca4c494a6d1a73648fd5813061 WatchSource:0}: Error finding container 478b55165f7266edb7fcc1290861751049be43ca4c494a6d1a73648fd5813061: Status 404 returned error can't find the container with id 478b55165f7266edb7fcc1290861751049be43ca4c494a6d1a73648fd5813061
Oct 30 20:01:09.613017 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.612980    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" event={"ID":"a88a932e-0217-4f8f-a956-54e5cef8e7b1","Type":"ContainerStarted","Data":"ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63"}
Oct 30 20:01:09.613017 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.613020    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" event={"ID":"a88a932e-0217-4f8f-a956-54e5cef8e7b1","Type":"ContainerStarted","Data":"478b55165f7266edb7fcc1290861751049be43ca4c494a6d1a73648fd5813061"}
Oct 30 20:01:09.614518 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.614497    4917 generic.go:356] "Generic (PLEG): container finished" podID="c8a955af-47dd-4248-b353-9b3ba02e9f2f" containerID="df7af9bb40ddedda2e6a0f6d1d9e117796c2609af02fb8d7876b466320ee6510" exitCode=0
Oct 30 20:01:09.614619 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.614558    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r" event={"ID":"c8a955af-47dd-4248-b353-9b3ba02e9f2f","Type":"ContainerDied","Data":"df7af9bb40ddedda2e6a0f6d1d9e117796c2609af02fb8d7876b466320ee6510"}
Oct 30 20:01:09.626427 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.626400    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:01:09.678898 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:01:09.678863    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:01:09.734603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:09.734559    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podStartSLOduration=1.734545225 podStartE2EDuration="1.734545225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:09.732687588 +0000 UTC m=+1165.897089231" watchObservedRunningTime="2026-10-30 20:01:09.734545225 +0000 UTC m=+1165.898946866"
Oct 30 20:01:10.039737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.039660    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:10.755052 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.755029    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:10.843928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.843899    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc\" (UniqueName: \"kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc\") pod \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") "
Oct 30 20:01:10.844090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.843940    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca\") pod \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") "
Oct 30 20:01:10.844293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.844250    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca" pod "c8a955af-47dd-4248-b353-9b3ba02e9f2f" (UID: "c8a955af-47dd-4248-b353-9b3ba02e9f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:10.845968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.845945    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc" pod "c8a955af-47dd-4248-b353-9b3ba02e9f2f" (UID: "c8a955af-47dd-4248-b353-9b3ba02e9f2f"). InnerVolumeSpecName "kube-api-access-rpzkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:10.944704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.944671    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca\") pod \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\" (UID: \"c8a955af-47dd-4248-b353-9b3ba02e9f2f\") "
Oct 30 20:01:10.944935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.944919    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpzkc\" (UniqueName: \"kubernetes.io/projected/c8a955af-47dd-4248-b353-9b3ba02e9f2f-kube-api-access-rpzkc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.944983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.944937    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.946735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:10.946708    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca" pod "c8a955af-47dd-4248-b353-9b3ba02e9f2f" (UID: "c8a955af-47dd-4248-b353-9b3ba02e9f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:11.045331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:11.045301    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a955af-47dd-4248-b353-9b3ba02e9f2f-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:11.627885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:11.627852    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"
Oct 30 20:01:11.628081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:11.627888    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r" event={"ID":"c8a955af-47dd-4248-b353-9b3ba02e9f2f","Type":"ContainerDied","Data":"a4a1faedfcfe1f5c062ba15d1854e3f0a61311d843f28862dc80bae378c38b71"}
Oct 30 20:01:11.628081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:11.627934    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a1faedfcfe1f5c062ba15d1854e3f0a61311d843f28862dc80bae378c38b71"
Oct 30 20:01:19.039509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:19.039475    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:19.052019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:19.051994    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:19.687542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:19.687515    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:01:21.498919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.498882    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"]
Oct 30 20:01:21.499514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.499494    4917 cpu_manager.go:414] "containerMap: removing container" podUID="c8a955af-47dd-4248-b353-9b3ba02e9f2f" containerName="revoke-keys"
Oct 30 20:01:21.499552 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.499516    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a955af-47dd-4248-b353-9b3ba02e9f2f" containerName="revoke-keys"
Oct 30 20:01:21.499680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.499668    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a955af-47dd-4248-b353-9b3ba02e9f2f" containerName="revoke-keys"
Oct 30 20:01:21.504157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.504135    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.557919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.557885    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"openshift-service-ca.crt\""
Oct 30 20:01:21.558101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.557978    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"default-dockercfg-qdntr\""
Oct 30 20:01:21.558101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.558043    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"kube-root-ca.crt\""
Oct 30 20:01:21.558101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.558073    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"premium-w6-f75272-kserve-self-signed-certs\""
Oct 30 20:01:21.560906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.560885    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"]
Oct 30 20:01:21.582162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.582139    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.582277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.582171    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.582277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.582211    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4b2l\" (UniqueName: \"kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.582395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.582337    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.582452 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.582397    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.582505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.582472    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.683601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.683564    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.683809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.683613    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.683809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.683658    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4b2l\" (UniqueName: \"kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.683809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.683698    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.683809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.683728    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.683809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.683804    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.684128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.684103    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.684252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.684216    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.684392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.684317    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.686409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.686388    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.686893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.686874    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.746113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.746085    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4b2l\" (UniqueName: \"kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l\") pod \"premium-w6-f75272-kserve-54749777dc-tsd5r\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") " pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:21.817085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:21.816993    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:22.213800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:22.213757    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"]
Oct 30 20:01:22.214826 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:22.214796    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05e3bdda_e405_4a41_97b0_9b67005898e8.slice/crio-4fff1cc50990c25b13900fdd7d6e2aed17955d3ec4f525209ccf658b427a0972 WatchSource:0}: Error finding container 4fff1cc50990c25b13900fdd7d6e2aed17955d3ec4f525209ccf658b427a0972: Status 404 returned error can't find the container with id 4fff1cc50990c25b13900fdd7d6e2aed17955d3ec4f525209ccf658b427a0972
Oct 30 20:01:22.695216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:22.695180    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" event={"ID":"05e3bdda-e405-4a41-97b0-9b67005898e8","Type":"ContainerStarted","Data":"cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc"}
Oct 30 20:01:22.695613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:22.695227    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" event={"ID":"05e3bdda-e405-4a41-97b0-9b67005898e8","Type":"ContainerStarted","Data":"4fff1cc50990c25b13900fdd7d6e2aed17955d3ec4f525209ccf658b427a0972"}
Oct 30 20:01:22.814333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:22.814281    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podStartSLOduration=1.814268599 podStartE2EDuration="1.814268599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:22.813088436 +0000 UTC m=+1178.977490080" watchObservedRunningTime="2026-10-30 20:01:22.814268599 +0000 UTC m=+1178.978670262"
Oct 30 20:01:22.817711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:22.817691    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:23.295100 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:23.295069    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:01:27.919712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:27.919669    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"]
Oct 30 20:01:27.923606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:27.923585    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:27.978607 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:27.978576    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:28.029127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.029094    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:28.029127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.029109    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:28.029127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.029115    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:28.031888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.031865    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"]
Oct 30 20:01:28.055221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.055196    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.055338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.055320    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.055388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.055360    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jgl\" (UniqueName: \"kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.156491 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.156455    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.156668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.156541    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jgl\" (UniqueName: \"kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.156668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.156611    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.157197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.157173    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.159021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.158997    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.219778 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.219737    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jgl\" (UniqueName: \"kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl\") pod \"maas-api-revoke-sub-48fa7c74-cm8d6\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.237584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.237557    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:28.611887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.611861    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"]
Oct 30 20:01:28.613653 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:28.613626    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff6f255_9917_4dc9_ba01_15b6e115e063.slice/crio-e66cf5199042477b68c895b4eefc1c7360188afb084af8a6cb4c041ff405b1d7 WatchSource:0}: Error finding container e66cf5199042477b68c895b4eefc1c7360188afb084af8a6cb4c041ff405b1d7: Status 404 returned error can't find the container with id e66cf5199042477b68c895b4eefc1c7360188afb084af8a6cb4c041ff405b1d7
Oct 30 20:01:28.729508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:28.729474    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6" event={"ID":"5ff6f255-9917-4dc9-ba01-15b6e115e063","Type":"ContainerStarted","Data":"e66cf5199042477b68c895b4eefc1c7360188afb084af8a6cb4c041ff405b1d7"}
Oct 30 20:01:29.737153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:29.737115    4917 generic.go:356] "Generic (PLEG): container finished" podID="5ff6f255-9917-4dc9-ba01-15b6e115e063" containerID="8a9e3e3f75e6b5b5557ce37fcf0ce3ebc8d9996ee16729e6dc3e61e09cd94ccd" exitCode=0
Oct 30 20:01:29.737568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:29.737198    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6" event={"ID":"5ff6f255-9917-4dc9-ba01-15b6e115e063","Type":"ContainerDied","Data":"8a9e3e3f75e6b5b5557ce37fcf0ce3ebc8d9996ee16729e6dc3e61e09cd94ccd"}
Oct 30 20:01:30.890296 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.890271    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:30.984788 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.984738    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca\") pod \"5ff6f255-9917-4dc9-ba01-15b6e115e063\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") "
Oct 30 20:01:30.984963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.984821    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca\") pod \"5ff6f255-9917-4dc9-ba01-15b6e115e063\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") "
Oct 30 20:01:30.984963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.984841    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl\" (UniqueName: \"kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl\") pod \"5ff6f255-9917-4dc9-ba01-15b6e115e063\" (UID: \"5ff6f255-9917-4dc9-ba01-15b6e115e063\") "
Oct 30 20:01:30.985139 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.985115    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca" pod "5ff6f255-9917-4dc9-ba01-15b6e115e063" (UID: "5ff6f255-9917-4dc9-ba01-15b6e115e063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:30.986975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.986944    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca" pod "5ff6f255-9917-4dc9-ba01-15b6e115e063" (UID: "5ff6f255-9917-4dc9-ba01-15b6e115e063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:30.987077 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:30.986972    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl" pod "5ff6f255-9917-4dc9-ba01-15b6e115e063" (UID: "5ff6f255-9917-4dc9-ba01-15b6e115e063"). InnerVolumeSpecName "kube-api-access-c9jgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:31.086238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.086206    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:31.086238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.086235    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff6f255-9917-4dc9-ba01-15b6e115e063-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:31.086238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.086246    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jgl\" (UniqueName: \"kubernetes.io/projected/5ff6f255-9917-4dc9-ba01-15b6e115e063-kube-api-access-c9jgl\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:31.751379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.751348    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"
Oct 30 20:01:31.751379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.751371    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6" event={"ID":"5ff6f255-9917-4dc9-ba01-15b6e115e063","Type":"ContainerDied","Data":"e66cf5199042477b68c895b4eefc1c7360188afb084af8a6cb4c041ff405b1d7"}
Oct 30 20:01:31.751593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.751445    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e66cf5199042477b68c895b4eefc1c7360188afb084af8a6cb4c041ff405b1d7"
Oct 30 20:01:31.817181 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.817150    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:31.830151 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:31.830126    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:32.769642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:32.769614    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:01:39.680908 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:01:39.680867    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:01:43.273253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.273219    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"]
Oct 30 20:01:43.273692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.273678    4917 cpu_manager.go:414] "containerMap: removing container" podUID="5ff6f255-9917-4dc9-ba01-15b6e115e063" containerName="revoke-keys"
Oct 30 20:01:43.273742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.273692    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff6f255-9917-4dc9-ba01-15b6e115e063" containerName="revoke-keys"
Oct 30 20:01:43.273806 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.273766    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff6f255-9917-4dc9-ba01-15b6e115e063" containerName="revoke-keys"
Oct 30 20:01:43.277172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.277156    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.334591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.334557    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"]
Oct 30 20:01:43.382049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.382004    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:43.382049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.382010    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:43.382049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.382011    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:43.382277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.382064    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:43.411237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.411202    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.411395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.411346    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.411436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.411402    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dv94\" (UniqueName: \"kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.512416 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.512378    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.512588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.512430    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dv94\" (UniqueName: \"kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.512588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.512504    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.513202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.513175    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.514937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.514909    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.574795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.574707    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dv94\" (UniqueName: \"kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94\") pod \"maas-api-revoke-sub-899e4914-7nth9\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.589588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.589552    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:43.968417 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:43.968384    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"]
Oct 30 20:01:43.970285 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:43.970254    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045d4486_1800_4482_bcdd_7a1dd2570fbe.slice/crio-d44c6f07fff6bda06350093b8eb5f5d0a194a9356b9881785dae2ecbdc0a6f49 WatchSource:0}: Error finding container d44c6f07fff6bda06350093b8eb5f5d0a194a9356b9881785dae2ecbdc0a6f49: Status 404 returned error can't find the container with id d44c6f07fff6bda06350093b8eb5f5d0a194a9356b9881785dae2ecbdc0a6f49
Oct 30 20:01:44.123737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:44.123706    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:01:44.133081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:44.133058    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:01:44.829103 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:44.829063    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9" event={"ID":"045d4486-1800-4482-bcdd-7a1dd2570fbe","Type":"ContainerStarted","Data":"b492e6f8ff3debe7bef119626d90f2e354da518d4902db353b5fcce2a29bf01e"}
Oct 30 20:01:44.829536 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:44.829151    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9" event={"ID":"045d4486-1800-4482-bcdd-7a1dd2570fbe","Type":"ContainerStarted","Data":"d44c6f07fff6bda06350093b8eb5f5d0a194a9356b9881785dae2ecbdc0a6f49"}
Oct 30 20:01:44.949631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:44.949571    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9" podStartSLOduration=1.94955409 podStartE2EDuration="1.94955409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:44.946216583 +0000 UTC m=+1201.110618224" watchObservedRunningTime="2026-10-30 20:01:44.94955409 +0000 UTC m=+1201.113955765"
Oct 30 20:01:45.836555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:45.836520    4917 generic.go:356] "Generic (PLEG): container finished" podID="045d4486-1800-4482-bcdd-7a1dd2570fbe" containerID="b492e6f8ff3debe7bef119626d90f2e354da518d4902db353b5fcce2a29bf01e" exitCode=0
Oct 30 20:01:45.836965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:45.836600    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9" event={"ID":"045d4486-1800-4482-bcdd-7a1dd2570fbe","Type":"ContainerDied","Data":"b492e6f8ff3debe7bef119626d90f2e354da518d4902db353b5fcce2a29bf01e"}
Oct 30 20:01:46.985431 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:46.985401    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:47.142750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.142662    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca\") pod \"045d4486-1800-4482-bcdd-7a1dd2570fbe\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") "
Oct 30 20:01:47.142750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.142730    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94\" (UniqueName: \"kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94\") pod \"045d4486-1800-4482-bcdd-7a1dd2570fbe\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") "
Oct 30 20:01:47.143012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.142895    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca\") pod \"045d4486-1800-4482-bcdd-7a1dd2570fbe\" (UID: \"045d4486-1800-4482-bcdd-7a1dd2570fbe\") "
Oct 30 20:01:47.143074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.143034    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca" pod "045d4486-1800-4482-bcdd-7a1dd2570fbe" (UID: "045d4486-1800-4482-bcdd-7a1dd2570fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:47.143210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.143192    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.144850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.144823    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca" pod "045d4486-1800-4482-bcdd-7a1dd2570fbe" (UID: "045d4486-1800-4482-bcdd-7a1dd2570fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:47.144964 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.144932    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94" pod "045d4486-1800-4482-bcdd-7a1dd2570fbe" (UID: "045d4486-1800-4482-bcdd-7a1dd2570fbe"). InnerVolumeSpecName "kube-api-access-9dv94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:47.244657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.244626    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dv94\" (UniqueName: \"kubernetes.io/projected/045d4486-1800-4482-bcdd-7a1dd2570fbe-kube-api-access-9dv94\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.244657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.244656    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045d4486-1800-4482-bcdd-7a1dd2570fbe-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.849399 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.849364    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"
Oct 30 20:01:47.849399 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.849390    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9" event={"ID":"045d4486-1800-4482-bcdd-7a1dd2570fbe","Type":"ContainerDied","Data":"d44c6f07fff6bda06350093b8eb5f5d0a194a9356b9881785dae2ecbdc0a6f49"}
Oct 30 20:01:47.849634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:47.849470    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44c6f07fff6bda06350093b8eb5f5d0a194a9356b9881785dae2ecbdc0a6f49"
Oct 30 20:01:48.855706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.855675    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"]
Oct 30 20:01:48.856111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.856103    4917 cpu_manager.go:414] "containerMap: removing container" podUID="045d4486-1800-4482-bcdd-7a1dd2570fbe" containerName="revoke-keys"
Oct 30 20:01:48.856158 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.856115    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045d4486-1800-4482-bcdd-7a1dd2570fbe" containerName="revoke-keys"
Oct 30 20:01:48.856213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.856202    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="045d4486-1800-4482-bcdd-7a1dd2570fbe" containerName="revoke-keys"
Oct 30 20:01:48.859355 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.859339    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:48.921634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.921603    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:48.960857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.960818    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:48.960857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.960857    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:48.961047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.961003    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwvl\" (UniqueName: \"kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:48.963945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.963924    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:48.964064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.963981    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:48.964064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.964020    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:48.966818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:48.966799    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"]
Oct 30 20:01:49.061747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.061712    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.061969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.061753    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.061969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.061854    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwvl\" (UniqueName: \"kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.062520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.062498    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.064173 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.064155    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.126963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.126885    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwvl\" (UniqueName: \"kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl\") pod \"maas-api-revoke-sub-c26bc6b8-vmdbt\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.172543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.172512    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:49.552723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.552696    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"]
Oct 30 20:01:49.554357 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:49.554332    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a4681c_bd23_4556_aae3_f2a6c6ef3653.slice/crio-dd0afabf598182825e9448758fc84532c006e9f68fb836d2fcd6d41eb0f5d0f5 WatchSource:0}: Error finding container dd0afabf598182825e9448758fc84532c006e9f68fb836d2fcd6d41eb0f5d0f5: Status 404 returned error can't find the container with id dd0afabf598182825e9448758fc84532c006e9f68fb836d2fcd6d41eb0f5d0f5
Oct 30 20:01:49.863191 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.863162    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt" event={"ID":"72a4681c-bd23-4556-aae3-f2a6c6ef3653","Type":"ContainerStarted","Data":"9fa3e236e9676818e2f003e699f5f815ac3b5849d1ec7897aafe6355e878c7b6"}
Oct 30 20:01:49.863509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.863215    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt" event={"ID":"72a4681c-bd23-4556-aae3-f2a6c6ef3653","Type":"ContainerStarted","Data":"dd0afabf598182825e9448758fc84532c006e9f68fb836d2fcd6d41eb0f5d0f5"}
Oct 30 20:01:49.993203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:49.993146    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt" podStartSLOduration=1.993133322 podStartE2EDuration="1.993133322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:49.991585766 +0000 UTC m=+1206.155987408" watchObservedRunningTime="2026-10-30 20:01:49.993133322 +0000 UTC m=+1206.157534964"
Oct 30 20:01:50.870530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:50.870491    4917 generic.go:356] "Generic (PLEG): container finished" podID="72a4681c-bd23-4556-aae3-f2a6c6ef3653" containerID="9fa3e236e9676818e2f003e699f5f815ac3b5849d1ec7897aafe6355e878c7b6" exitCode=0
Oct 30 20:01:50.870949 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:50.870552    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt" event={"ID":"72a4681c-bd23-4556-aae3-f2a6c6ef3653","Type":"ContainerDied","Data":"9fa3e236e9676818e2f003e699f5f815ac3b5849d1ec7897aafe6355e878c7b6"}
Oct 30 20:01:52.030844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.030817    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:52.095077 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.095036    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl\" (UniqueName: \"kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl\") pod \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") "
Oct 30 20:01:52.095238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.095129    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca\") pod \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") "
Oct 30 20:01:52.095340 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.095322    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca\") pod \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\" (UID: \"72a4681c-bd23-4556-aae3-f2a6c6ef3653\") "
Oct 30 20:01:52.095511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.095488    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca" pod "72a4681c-bd23-4556-aae3-f2a6c6ef3653" (UID: "72a4681c-bd23-4556-aae3-f2a6c6ef3653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:52.095823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.095800    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.097376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.097354    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca" pod "72a4681c-bd23-4556-aae3-f2a6c6ef3653" (UID: "72a4681c-bd23-4556-aae3-f2a6c6ef3653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:52.097460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.097436    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl" pod "72a4681c-bd23-4556-aae3-f2a6c6ef3653" (UID: "72a4681c-bd23-4556-aae3-f2a6c6ef3653"). InnerVolumeSpecName "kube-api-access-qpwvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:52.197045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.197010    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a4681c-bd23-4556-aae3-f2a6c6ef3653-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.197045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.197039    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwvl\" (UniqueName: \"kubernetes.io/projected/72a4681c-bd23-4556-aae3-f2a6c6ef3653-kube-api-access-qpwvl\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.882891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.882849    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt" event={"ID":"72a4681c-bd23-4556-aae3-f2a6c6ef3653","Type":"ContainerDied","Data":"dd0afabf598182825e9448758fc84532c006e9f68fb836d2fcd6d41eb0f5d0f5"}
Oct 30 20:01:52.882891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.882871    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"
Oct 30 20:01:52.883110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:52.882907    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd0afabf598182825e9448758fc84532c006e9f68fb836d2fcd6d41eb0f5d0f5"
Oct 30 20:01:54.941394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.941318    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"]
Oct 30 20:01:54.941816 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.941801    4917 cpu_manager.go:414] "containerMap: removing container" podUID="72a4681c-bd23-4556-aae3-f2a6c6ef3653" containerName="revoke-keys"
Oct 30 20:01:54.941816 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.941816    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a4681c-bd23-4556-aae3-f2a6c6ef3653" containerName="revoke-keys"
Oct 30 20:01:54.941914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.941897    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a4681c-bd23-4556-aae3-f2a6c6ef3653" containerName="revoke-keys"
Oct 30 20:01:54.945252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.945233    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:54.999263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.999224    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:54.999460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:54.999430    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:55.025022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.024989    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.025187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.025044    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.025187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.025079    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhpm\" (UniqueName: \"kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.049863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.049833    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:55.050019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.049945    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:55.052797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.052765    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"]
Oct 30 20:01:55.126425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.126386    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.126599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.126436    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.126599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.126483    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhpm\" (UniqueName: \"kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.127092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.127074    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.128861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.128835    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.188589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.188557    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhpm\" (UniqueName: \"kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm\") pod \"maas-api-revoke-sub-0c314d64-dc7vz\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.258078 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.257986    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:55.632025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.631999    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"]
Oct 30 20:01:55.633357 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:01:55.633327    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6efee9_7d79_4460_92a1_79c622e7c3c1.slice/crio-29f9fbdff918bba503836dabbac9d33dfd4d182a758d8a62577aa72ea454a30b WatchSource:0}: Error finding container 29f9fbdff918bba503836dabbac9d33dfd4d182a758d8a62577aa72ea454a30b: Status 404 returned error can't find the container with id 29f9fbdff918bba503836dabbac9d33dfd4d182a758d8a62577aa72ea454a30b
Oct 30 20:01:55.902748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:55.902663    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz" event={"ID":"ff6efee9-7d79-4460-92a1-79c622e7c3c1","Type":"ContainerStarted","Data":"29f9fbdff918bba503836dabbac9d33dfd4d182a758d8a62577aa72ea454a30b"}
Oct 30 20:01:56.913296 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:56.913195    4917 generic.go:356] "Generic (PLEG): container finished" podID="ff6efee9-7d79-4460-92a1-79c622e7c3c1" containerID="00c648f7b2789f327c0ecd457f571683989c110094e926fcd2ab8a0e71058a4f" exitCode=0
Oct 30 20:01:56.913296 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:56.913265    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz" event={"ID":"ff6efee9-7d79-4460-92a1-79c622e7c3c1","Type":"ContainerDied","Data":"00c648f7b2789f327c0ecd457f571683989c110094e926fcd2ab8a0e71058a4f"}
Oct 30 20:01:58.061610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.061585    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:58.157792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.157743    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca\") pod \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") "
Oct 30 20:01:58.157989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.157849    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca\") pod \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") "
Oct 30 20:01:58.158057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.157997    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm\" (UniqueName: \"kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm\") pod \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\" (UID: \"ff6efee9-7d79-4460-92a1-79c622e7c3c1\") "
Oct 30 20:01:58.158164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.158142    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca" pod "ff6efee9-7d79-4460-92a1-79c622e7c3c1" (UID: "ff6efee9-7d79-4460-92a1-79c622e7c3c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:58.158366 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.158338    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:58.160013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.159983    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm" pod "ff6efee9-7d79-4460-92a1-79c622e7c3c1" (UID: "ff6efee9-7d79-4460-92a1-79c622e7c3c1"). InnerVolumeSpecName "kube-api-access-mdhpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:58.160100 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.160024    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca" pod "ff6efee9-7d79-4460-92a1-79c622e7c3c1" (UID: "ff6efee9-7d79-4460-92a1-79c622e7c3c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:58.259250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.259216    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdhpm\" (UniqueName: \"kubernetes.io/projected/ff6efee9-7d79-4460-92a1-79c622e7c3c1-kube-api-access-mdhpm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:58.259250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.259249    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6efee9-7d79-4460-92a1-79c622e7c3c1-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:58.926901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.926866    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz" event={"ID":"ff6efee9-7d79-4460-92a1-79c622e7c3c1","Type":"ContainerDied","Data":"29f9fbdff918bba503836dabbac9d33dfd4d182a758d8a62577aa72ea454a30b"}
Oct 30 20:01:58.926901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.926896    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"
Oct 30 20:01:58.927175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:58.926919    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f9fbdff918bba503836dabbac9d33dfd4d182a758d8a62577aa72ea454a30b"
Oct 30 20:01:59.685733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.685701    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"]
Oct 30 20:01:59.686173 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.686159    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ff6efee9-7d79-4460-92a1-79c622e7c3c1" containerName="revoke-keys"
Oct 30 20:01:59.686216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.686173    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6efee9-7d79-4460-92a1-79c622e7c3c1" containerName="revoke-keys"
Oct 30 20:01:59.686271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.686262    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6efee9-7d79-4460-92a1-79c622e7c3c1" containerName="revoke-keys"
Oct 30 20:01:59.689208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.689188    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.750540 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.750510    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:59.750755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.750645    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:59.750755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.750676    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:59.774635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.774603    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qsq\" (UniqueName: \"kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.774852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.774705    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.774852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.774735    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.796586 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.796557    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:59.799461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.799439    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"]
Oct 30 20:01:59.876128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.876092    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qsq\" (UniqueName: \"kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.876333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.876158    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.876333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.876186    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.876863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.876838    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.878684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.878659    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:01:59.938965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:01:59.938900    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qsq\" (UniqueName: \"kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq\") pod \"maas-api-revoke-sub-458882e7-rpspn\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:02:00.002927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:00.002623    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:02:00.377277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:00.377247    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"]
Oct 30 20:02:00.378302 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:00.378270    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079348fe_cdd8_424e_aa73_7a655fbd13fa.slice/crio-b7f6610857f79535f1140d999f453273ebadf4b3e81ab4aafa536b1ded97de34 WatchSource:0}: Error finding container b7f6610857f79535f1140d999f453273ebadf4b3e81ab4aafa536b1ded97de34: Status 404 returned error can't find the container with id b7f6610857f79535f1140d999f453273ebadf4b3e81ab4aafa536b1ded97de34
Oct 30 20:02:00.939317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:00.939281    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn" event={"ID":"079348fe-cdd8-424e-aa73-7a655fbd13fa","Type":"ContainerStarted","Data":"0c78bc7df57e041bbf414a874c3922f701c24818546112a266c7a2ed752d7e4d"}
Oct 30 20:02:00.939723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:00.939349    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn" event={"ID":"079348fe-cdd8-424e-aa73-7a655fbd13fa","Type":"ContainerStarted","Data":"b7f6610857f79535f1140d999f453273ebadf4b3e81ab4aafa536b1ded97de34"}
Oct 30 20:02:01.058635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:01.058585    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn" podStartSLOduration=2.058570607 podStartE2EDuration="2.058570607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:01.057736958 +0000 UTC m=+1217.222138602" watchObservedRunningTime="2026-10-30 20:02:01.058570607 +0000 UTC m=+1217.222972248"
Oct 30 20:02:01.947577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:01.947539    4917 generic.go:356] "Generic (PLEG): container finished" podID="079348fe-cdd8-424e-aa73-7a655fbd13fa" containerID="0c78bc7df57e041bbf414a874c3922f701c24818546112a266c7a2ed752d7e4d" exitCode=0
Oct 30 20:02:01.947993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:01.947628    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn" event={"ID":"079348fe-cdd8-424e-aa73-7a655fbd13fa","Type":"ContainerDied","Data":"0c78bc7df57e041bbf414a874c3922f701c24818546112a266c7a2ed752d7e4d"}
Oct 30 20:02:02.435460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:02.435419    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:02:03.097535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.097504    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:02:03.210734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.210646    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq\" (UniqueName: \"kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq\") pod \"079348fe-cdd8-424e-aa73-7a655fbd13fa\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") "
Oct 30 20:02:03.210734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.210718    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca\") pod \"079348fe-cdd8-424e-aa73-7a655fbd13fa\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") "
Oct 30 20:02:03.210989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.210818    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca\") pod \"079348fe-cdd8-424e-aa73-7a655fbd13fa\" (UID: \"079348fe-cdd8-424e-aa73-7a655fbd13fa\") "
Oct 30 20:02:03.211124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.211095    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca" pod "079348fe-cdd8-424e-aa73-7a655fbd13fa" (UID: "079348fe-cdd8-424e-aa73-7a655fbd13fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:03.212888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.212858    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq" pod "079348fe-cdd8-424e-aa73-7a655fbd13fa" (UID: "079348fe-cdd8-424e-aa73-7a655fbd13fa"). InnerVolumeSpecName "kube-api-access-l5qsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:03.212888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.212870    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca" pod "079348fe-cdd8-424e-aa73-7a655fbd13fa" (UID: "079348fe-cdd8-424e-aa73-7a655fbd13fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:03.312690 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.312648    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.312690 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.312694    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qsq\" (UniqueName: \"kubernetes.io/projected/079348fe-cdd8-424e-aa73-7a655fbd13fa-kube-api-access-l5qsq\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.312927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.312712    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079348fe-cdd8-424e-aa73-7a655fbd13fa-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.962501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.962474    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"
Oct 30 20:02:03.962649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.962510    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn" event={"ID":"079348fe-cdd8-424e-aa73-7a655fbd13fa","Type":"ContainerDied","Data":"b7f6610857f79535f1140d999f453273ebadf4b3e81ab4aafa536b1ded97de34"}
Oct 30 20:02:03.962649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:03.962558    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7f6610857f79535f1140d999f453273ebadf4b3e81ab4aafa536b1ded97de34"
Oct 30 20:02:09.679406 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:02:09.679361    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:02:14.340492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.340452    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"]
Oct 30 20:02:14.341172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.341151    4917 cpu_manager.go:414] "containerMap: removing container" podUID="079348fe-cdd8-424e-aa73-7a655fbd13fa" containerName="revoke-keys"
Oct 30 20:02:14.341274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.341175    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079348fe-cdd8-424e-aa73-7a655fbd13fa" containerName="revoke-keys"
Oct 30 20:02:14.341360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.341346    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="079348fe-cdd8-424e-aa73-7a655fbd13fa" containerName="revoke-keys"
Oct 30 20:02:14.344718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.344696    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.408963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.408929    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"]
Oct 30 20:02:14.426208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.426170    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.426456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.426243    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99kn\" (UniqueName: \"kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.426456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.426332    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.448511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.448476    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:14.448511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.448470    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:14.448713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.448474    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:14.448713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.448606    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:14.527784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.527742    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.527994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.527930    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.528067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.528006    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99kn\" (UniqueName: \"kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.528566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.528544    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.530243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.530225    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.590710 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.590630    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99kn\" (UniqueName: \"kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn\") pod \"maas-api-revoke-sub-f666521b-xx7d7\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:14.657716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:14.657680    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:15.034179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:15.034151    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"]
Oct 30 20:02:15.036621 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:15.036593    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b33878_846c_433e_8263_91d56276a0bd.slice/crio-3dced0d8e6e7dbe407ff989c508a6efdb1dd2438542d7153d06c3c6b1c81acec WatchSource:0}: Error finding container 3dced0d8e6e7dbe407ff989c508a6efdb1dd2438542d7153d06c3c6b1c81acec: Status 404 returned error can't find the container with id 3dced0d8e6e7dbe407ff989c508a6efdb1dd2438542d7153d06c3c6b1c81acec
Oct 30 20:02:16.042043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:16.042008    4917 generic.go:356] "Generic (PLEG): container finished" podID="19b33878-846c-433e-8263-91d56276a0bd" containerID="76e42818c8ed3464d0fbdb11c57f9f551ed8bc786866bd26d599001f51155c2f" exitCode=0
Oct 30 20:02:16.042459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:16.042102    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7" event={"ID":"19b33878-846c-433e-8263-91d56276a0bd","Type":"ContainerDied","Data":"76e42818c8ed3464d0fbdb11c57f9f551ed8bc786866bd26d599001f51155c2f"}
Oct 30 20:02:16.042459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:16.042152    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7" event={"ID":"19b33878-846c-433e-8263-91d56276a0bd","Type":"ContainerStarted","Data":"3dced0d8e6e7dbe407ff989c508a6efdb1dd2438542d7153d06c3c6b1c81acec"}
Oct 30 20:02:17.202897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.202870    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:17.358081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.358004    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca\") pod \"19b33878-846c-433e-8263-91d56276a0bd\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") "
Oct 30 20:02:17.358237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.358170    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca\") pod \"19b33878-846c-433e-8263-91d56276a0bd\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") "
Oct 30 20:02:17.358284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.358253    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn\" (UniqueName: \"kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn\") pod \"19b33878-846c-433e-8263-91d56276a0bd\" (UID: \"19b33878-846c-433e-8263-91d56276a0bd\") "
Oct 30 20:02:17.358544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.358515    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca" pod "19b33878-846c-433e-8263-91d56276a0bd" (UID: "19b33878-846c-433e-8263-91d56276a0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:17.360306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.360272    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca" pod "19b33878-846c-433e-8263-91d56276a0bd" (UID: "19b33878-846c-433e-8263-91d56276a0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:17.360390 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.360346    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn" pod "19b33878-846c-433e-8263-91d56276a0bd" (UID: "19b33878-846c-433e-8263-91d56276a0bd"). InnerVolumeSpecName "kube-api-access-s99kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:17.459212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.459179    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33878-846c-433e-8263-91d56276a0bd-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.459212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.459213    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33878-846c-433e-8263-91d56276a0bd-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.459437 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:17.459230    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99kn\" (UniqueName: \"kubernetes.io/projected/19b33878-846c-433e-8263-91d56276a0bd-kube-api-access-s99kn\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:18.055385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:18.055356    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"
Oct 30 20:02:18.055550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:18.055360    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7" event={"ID":"19b33878-846c-433e-8263-91d56276a0bd","Type":"ContainerDied","Data":"3dced0d8e6e7dbe407ff989c508a6efdb1dd2438542d7153d06c3c6b1c81acec"}
Oct 30 20:02:18.055550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:18.055484    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dced0d8e6e7dbe407ff989c508a6efdb1dd2438542d7153d06c3c6b1c81acec"
Oct 30 20:02:20.549862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.549825    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"]
Oct 30 20:02:20.550322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.550307    4917 cpu_manager.go:414] "containerMap: removing container" podUID="19b33878-846c-433e-8263-91d56276a0bd" containerName="revoke-keys"
Oct 30 20:02:20.550361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.550323    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b33878-846c-433e-8263-91d56276a0bd" containerName="revoke-keys"
Oct 30 20:02:20.550442 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.550433    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b33878-846c-433e-8263-91d56276a0bd" containerName="revoke-keys"
Oct 30 20:02:20.553859 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.553842    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.614067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.614026    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"]
Oct 30 20:02:20.659670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.659636    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:20.659670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.659657    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:20.659670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.659670    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:20.659964 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.659688    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:20.691511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.691474    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.691693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.691594    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqb82\" (UniqueName: \"kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.691693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.691631    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.793358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.793318    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.793573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.793507    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqb82\" (UniqueName: \"kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.793573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.793555    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.794444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.794411    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.797435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.797406    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.855929 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.855848    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqb82\" (UniqueName: \"kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82\") pod \"maas-api-revoke-sub-8d858750-pgxb8\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:20.867496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:20.867471    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:21.268525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:21.268452    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"]
Oct 30 20:02:22.083363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:22.083317    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8" event={"ID":"17ceb1d2-8b85-41de-a302-2e5abe7a467a","Type":"ContainerStarted","Data":"1c16479dbec9205c5f289d592f77da682d552d3a374907565094ba9ca32d21d2"}
Oct 30 20:02:22.083917 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:22.083373    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8" event={"ID":"17ceb1d2-8b85-41de-a302-2e5abe7a467a","Type":"ContainerStarted","Data":"ef016c71bf9b5fdf30ba7f99703eaab65f9e1e751b494f110f16ab24f83ded2c"}
Oct 30 20:02:22.203186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:22.203117    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8" podStartSLOduration=2.203099683 podStartE2EDuration="2.203099683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.201044399 +0000 UTC m=+1238.365446040" watchObservedRunningTime="2026-10-30 20:02:22.203099683 +0000 UTC m=+1238.367501331"
Oct 30 20:02:23.091512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:23.091475    4917 generic.go:356] "Generic (PLEG): container finished" podID="17ceb1d2-8b85-41de-a302-2e5abe7a467a" containerID="1c16479dbec9205c5f289d592f77da682d552d3a374907565094ba9ca32d21d2" exitCode=0
Oct 30 20:02:23.091957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:23.091562    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8" event={"ID":"17ceb1d2-8b85-41de-a302-2e5abe7a467a","Type":"ContainerDied","Data":"1c16479dbec9205c5f289d592f77da682d552d3a374907565094ba9ca32d21d2"}
Oct 30 20:02:24.245743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.245716    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:24.337121 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.337084    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82\" (UniqueName: \"kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82\") pod \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") "
Oct 30 20:02:24.337295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.337145    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca\") pod \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") "
Oct 30 20:02:24.337295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.337230    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca\") pod \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\" (UID: \"17ceb1d2-8b85-41de-a302-2e5abe7a467a\") "
Oct 30 20:02:24.337598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.337572    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca" pod "17ceb1d2-8b85-41de-a302-2e5abe7a467a" (UID: "17ceb1d2-8b85-41de-a302-2e5abe7a467a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.339292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.339265    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca" pod "17ceb1d2-8b85-41de-a302-2e5abe7a467a" (UID: "17ceb1d2-8b85-41de-a302-2e5abe7a467a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:24.339402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.339265    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82" pod "17ceb1d2-8b85-41de-a302-2e5abe7a467a" (UID: "17ceb1d2-8b85-41de-a302-2e5abe7a467a"). InnerVolumeSpecName "kube-api-access-nqb82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.438310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.438276    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.438310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.438312    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqb82\" (UniqueName: \"kubernetes.io/projected/17ceb1d2-8b85-41de-a302-2e5abe7a467a-kube-api-access-nqb82\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.438494 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:24.438328    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ceb1d2-8b85-41de-a302-2e5abe7a467a-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:25.104729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:25.104701    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"
Oct 30 20:02:25.104923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:25.104734    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8" event={"ID":"17ceb1d2-8b85-41de-a302-2e5abe7a467a","Type":"ContainerDied","Data":"ef016c71bf9b5fdf30ba7f99703eaab65f9e1e751b494f110f16ab24f83ded2c"}
Oct 30 20:02:25.104923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:25.104802    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef016c71bf9b5fdf30ba7f99703eaab65f9e1e751b494f110f16ab24f83ded2c"
Oct 30 20:02:26.237202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.237165    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"]
Oct 30 20:02:26.237750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.237732    4917 cpu_manager.go:414] "containerMap: removing container" podUID="17ceb1d2-8b85-41de-a302-2e5abe7a467a" containerName="revoke-keys"
Oct 30 20:02:26.237884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.237752    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ceb1d2-8b85-41de-a302-2e5abe7a467a" containerName="revoke-keys"
Oct 30 20:02:26.237884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.237867    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ceb1d2-8b85-41de-a302-2e5abe7a467a" containerName="revoke-keys"
Oct 30 20:02:26.241141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.241124    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.298385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.298334    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:26.348906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.348869    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:26.348906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.348890    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:26.348906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.348900    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:26.352031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.352009    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"]
Oct 30 20:02:26.356043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.356016    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqvc\" (UniqueName: \"kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.356174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.356155    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.356269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.356201    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.456844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.456799    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.457047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.456869    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.457047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.456946    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqvc\" (UniqueName: \"kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.457603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.457577    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.459350 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.459323    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.519825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.519735    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqvc\" (UniqueName: \"kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc\") pod \"maas-api-revoke-sub-36ce5cdc-pnswb\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.555723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.555690    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:26.930946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:26.930918    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"]
Oct 30 20:02:26.931760 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:26.931736    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c04c20c_1eb6_4b9a_8378_45c2b2fa14e0.slice/crio-a82bc7d15d8884d419fe7ca33c4756c36f782b95c11c2756a54040cb90252456 WatchSource:0}: Error finding container a82bc7d15d8884d419fe7ca33c4756c36f782b95c11c2756a54040cb90252456: Status 404 returned error can't find the container with id a82bc7d15d8884d419fe7ca33c4756c36f782b95c11c2756a54040cb90252456
Oct 30 20:02:27.126984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:27.126951    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb" event={"ID":"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0","Type":"ContainerStarted","Data":"a82bc7d15d8884d419fe7ca33c4756c36f782b95c11c2756a54040cb90252456"}
Oct 30 20:02:28.134193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:28.134155    4917 generic.go:356] "Generic (PLEG): container finished" podID="4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" containerID="2221b186a5245fa32bb38d29c65812d8c87fbf75e37960036d7e9324cde90567" exitCode=0
Oct 30 20:02:28.134582 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:28.134216    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb" event={"ID":"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0","Type":"ContainerDied","Data":"2221b186a5245fa32bb38d29c65812d8c87fbf75e37960036d7e9324cde90567"}
Oct 30 20:02:29.284598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.284574    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:29.386461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.386431    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca\") pod \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") "
Oct 30 20:02:29.386597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.386519    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc\" (UniqueName: \"kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc\") pod \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") "
Oct 30 20:02:29.386597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.386538    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca\") pod \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\" (UID: \"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0\") "
Oct 30 20:02:29.386980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.386956    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca" pod "4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" (UID: "4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:29.388572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.388544    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca" pod "4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" (UID: "4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:29.388729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.388706    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc" pod "4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" (UID: "4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0"). InnerVolumeSpecName "kube-api-access-xxqvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:29.488341 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.488309    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.488511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.488358    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqvc\" (UniqueName: \"kubernetes.io/projected/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-kube-api-access-xxqvc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.488511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:29.488370    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:30.146993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.146961    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"
Oct 30 20:02:30.147200 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.147004    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb" event={"ID":"4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0","Type":"ContainerDied","Data":"a82bc7d15d8884d419fe7ca33c4756c36f782b95c11c2756a54040cb90252456"}
Oct 30 20:02:30.147200 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.147051    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82bc7d15d8884d419fe7ca33c4756c36f782b95c11c2756a54040cb90252456"
Oct 30 20:02:30.401535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.401441    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"]
Oct 30 20:02:30.401955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.401916    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" containerName="revoke-keys"
Oct 30 20:02:30.401955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.401928    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" containerName="revoke-keys"
Oct 30 20:02:30.402034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.402009    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" containerName="revoke-keys"
Oct 30 20:02:30.405505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.405488    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.462223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.462194    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:30.480886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.480853    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:30.480886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.480864    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:30.480886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.480866    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:30.484085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.484062    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"]
Oct 30 20:02:30.497916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.497892    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.498045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.497940    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm79s\" (UniqueName: \"kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.498150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.498127    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.599491 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.599447    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.599682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.599504    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm79s\" (UniqueName: \"kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.599682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.599586    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.600280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.600255    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.601960 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.601939    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.662700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.662626    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm79s\" (UniqueName: \"kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s\") pod \"maas-api-revoke-sub-6763c652-fgqbc\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:30.719015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:30.718975    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:31.098629 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:31.098601    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"]
Oct 30 20:02:31.099670 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:31.099635    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18d0377_d700_40e1_a8ea_12c63f50f64c.slice/crio-9af4ee728da5c0a89464640e23cbd525a28f546cf9189a145d2ac17414e95ce2 WatchSource:0}: Error finding container 9af4ee728da5c0a89464640e23cbd525a28f546cf9189a145d2ac17414e95ce2: Status 404 returned error can't find the container with id 9af4ee728da5c0a89464640e23cbd525a28f546cf9189a145d2ac17414e95ce2
Oct 30 20:02:31.155044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:31.155012    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc" event={"ID":"c18d0377-d700-40e1-a8ea-12c63f50f64c","Type":"ContainerStarted","Data":"9af4ee728da5c0a89464640e23cbd525a28f546cf9189a145d2ac17414e95ce2"}
Oct 30 20:02:32.163299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:32.163207    4917 generic.go:356] "Generic (PLEG): container finished" podID="c18d0377-d700-40e1-a8ea-12c63f50f64c" containerID="f6fe0265e3a6bf3024c5e6738fda2b4ef1b8626f232bfcf71fe60566b37c5419" exitCode=0
Oct 30 20:02:32.163299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:32.163251    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc" event={"ID":"c18d0377-d700-40e1-a8ea-12c63f50f64c","Type":"ContainerDied","Data":"f6fe0265e3a6bf3024c5e6738fda2b4ef1b8626f232bfcf71fe60566b37c5419"}
Oct 30 20:02:33.321453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.321422    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:33.426152 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.426069    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca\") pod \"c18d0377-d700-40e1-a8ea-12c63f50f64c\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") "
Oct 30 20:02:33.426292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.426171    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s\" (UniqueName: \"kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s\") pod \"c18d0377-d700-40e1-a8ea-12c63f50f64c\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") "
Oct 30 20:02:33.426292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.426217    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca\") pod \"c18d0377-d700-40e1-a8ea-12c63f50f64c\" (UID: \"c18d0377-d700-40e1-a8ea-12c63f50f64c\") "
Oct 30 20:02:33.426654 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.426624    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca" pod "c18d0377-d700-40e1-a8ea-12c63f50f64c" (UID: "c18d0377-d700-40e1-a8ea-12c63f50f64c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:33.428205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.428171    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca" pod "c18d0377-d700-40e1-a8ea-12c63f50f64c" (UID: "c18d0377-d700-40e1-a8ea-12c63f50f64c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:33.428308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.428276    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s" pod "c18d0377-d700-40e1-a8ea-12c63f50f64c" (UID: "c18d0377-d700-40e1-a8ea-12c63f50f64c"). InnerVolumeSpecName "kube-api-access-jm79s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:33.527645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.527604    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm79s\" (UniqueName: \"kubernetes.io/projected/c18d0377-d700-40e1-a8ea-12c63f50f64c-kube-api-access-jm79s\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.527645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.527636    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.527645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:33.527646    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18d0377-d700-40e1-a8ea-12c63f50f64c-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:34.178791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.178739    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"
Oct 30 20:02:34.178978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.178796    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc" event={"ID":"c18d0377-d700-40e1-a8ea-12c63f50f64c","Type":"ContainerDied","Data":"9af4ee728da5c0a89464640e23cbd525a28f546cf9189a145d2ac17414e95ce2"}
Oct 30 20:02:34.178978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.178853    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af4ee728da5c0a89464640e23cbd525a28f546cf9189a145d2ac17414e95ce2"
Oct 30 20:02:34.921765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.921731    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"]
Oct 30 20:02:34.922197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.922174    4917 cpu_manager.go:414] "containerMap: removing container" podUID="c18d0377-d700-40e1-a8ea-12c63f50f64c" containerName="revoke-keys"
Oct 30 20:02:34.922197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.922190    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18d0377-d700-40e1-a8ea-12c63f50f64c" containerName="revoke-keys"
Oct 30 20:02:34.922337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.922298    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18d0377-d700-40e1-a8ea-12c63f50f64c" containerName="revoke-keys"
Oct 30 20:02:34.925635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:34.925620    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.042940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.042906    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdp48\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.042959    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043119    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043162    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043233    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043292    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043321    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043368    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.043475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.043400    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.144225 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.144175    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdp48\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.144420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.144300    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"]
Oct 30 20:02:35.144795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.144740    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.144911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.144855    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.144911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.144891    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145192    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145249    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145252    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145307    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145311    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145366    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145415    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145443    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.145995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.145759    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.146749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.146161    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.151080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.147965    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.151080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.148476    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.206502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.206469    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdp48\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.207383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.207358    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pqs8j\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.238823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.238796    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:35.673059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.673028    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"]
Oct 30 20:02:35.673654 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:35.673621    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0df23aa_527b_4f03_b269_ff5acf36726d.slice/crio-cf49d7b87fce95293d83a8ad2a1995476f58ec9a8f46ce212d407eb8f2755924 WatchSource:0}: Error finding container cf49d7b87fce95293d83a8ad2a1995476f58ec9a8f46ce212d407eb8f2755924: Status 404 returned error can't find the container with id cf49d7b87fce95293d83a8ad2a1995476f58ec9a8f46ce212d407eb8f2755924
Oct 30 20:02:35.676215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.676184    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:35.676324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.676265    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:35.676324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:35.676310    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:36.198103 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:36.198064    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" event={"ID":"c0df23aa-527b-4f03-b269-ff5acf36726d","Type":"ContainerStarted","Data":"b1b4b578e2e1f45e6092cdb1badf249c279921785b1b3e4ac9b7041d2109e376"}
Oct 30 20:02:36.198713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:36.198686    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" event={"ID":"c0df23aa-527b-4f03-b269-ff5acf36726d","Type":"ContainerStarted","Data":"cf49d7b87fce95293d83a8ad2a1995476f58ec9a8f46ce212d407eb8f2755924"}
Oct 30 20:02:36.239606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:36.239574    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:36.244057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:36.243658    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:36.244057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:36.243757    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:36.337842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:36.337789    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podStartSLOduration=2.337762262 podStartE2EDuration="2.337762262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:36.334361476 +0000 UTC m=+1252.498763134" watchObservedRunningTime="2026-10-30 20:02:36.337762262 +0000 UTC m=+1252.502163903"
Oct 30 20:02:37.239823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:37.239790    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:37.240210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:37.239860    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:38.239481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:38.239448    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:38.239671 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:38.239501    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:39.239750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.239715    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:39.240156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.239807    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:39.382258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.382224    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"]
Oct 30 20:02:39.386092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.386070    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.440395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.440358    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"default-dockercfg-wdcjk\""
Oct 30 20:02:39.440568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.440408    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"distinct-w0-779921-kserve-self-signed-certs\""
Oct 30 20:02:39.486542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.486512    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"openshift-service-ca.crt\""
Oct 30 20:02:39.490669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.490592    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"kube-root-ca.crt\""
Oct 30 20:02:39.494280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.494259    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"]
Oct 30 20:02:39.497514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.497494    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lp52\" (UniqueName: \"kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.497577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.497523    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.497577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.497560    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.497752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.497731    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.497810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.497763    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.497880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.497862    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599405    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599457    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599505    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599539    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lp52\" (UniqueName: \"kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599688    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599813    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.599960 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599944    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.600001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.599961    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.600132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.600112    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.602232 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.602207    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.602630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.602609    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.671978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.671942    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lp52\" (UniqueName: \"kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52\") pod \"distinct-w0-779921-kserve-9887bf5f6-v6z52\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") " pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:39.695574 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:02:39.695530    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:02:39.699322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:39.699289    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:40.128512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:40.128480    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"]
Oct 30 20:02:40.129521 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:40.129494    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7372bcc3_6279_43ed_94b1_77889b4d9435.slice/crio-b84bb257c893dde586da1b6e9eb139847167455d3a567b4a0ecfaa116c8ee9d3 WatchSource:0}: Error finding container b84bb257c893dde586da1b6e9eb139847167455d3a567b4a0ecfaa116c8ee9d3: Status 404 returned error can't find the container with id b84bb257c893dde586da1b6e9eb139847167455d3a567b4a0ecfaa116c8ee9d3
Oct 30 20:02:40.225164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:40.225133    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" event={"ID":"7372bcc3-6279-43ed-94b1-77889b4d9435","Type":"ContainerStarted","Data":"b84bb257c893dde586da1b6e9eb139847167455d3a567b4a0ecfaa116c8ee9d3"}
Oct 30 20:02:40.239705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:40.239675    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:40.239863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:40.239726    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:41.233385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:41.233340    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" event={"ID":"7372bcc3-6279-43ed-94b1-77889b4d9435","Type":"ContainerStarted","Data":"f7c63b058a446a6babaaa3b3c846eb3afd770b0421155e2b961dd195738f630c"}
Oct 30 20:02:41.239825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:41.239794    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:41.239998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:41.239845    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:41.358795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:41.353983    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podStartSLOduration=2.353964048 podStartE2EDuration="2.353964048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:41.351480167 +0000 UTC m=+1257.515881811" watchObservedRunningTime="2026-10-30 20:02:41.353964048 +0000 UTC m=+1257.518365691"
Oct 30 20:02:41.571846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:41.571414    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:02:41.700159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:41.700122    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:42.239684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:42.239646    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:42.239854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:42.239709    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:43.239292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:43.239254    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:02:43.239701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:43.239319    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:02:44.311424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:44.311387    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:44.312005 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:44.311979    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:44.313035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:44.313012    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:02:45.737088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.737052    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"]
Oct 30 20:02:45.742067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.742038    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.795573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.795539    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d67ea9-openshift-default-dockercfg-t5857\""
Oct 30 20:02:45.851389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.851345    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"]
Oct 30 20:02:45.869715 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.869671    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.869715 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.869732    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.869896    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.869933    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.869974    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.870079    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8c7w\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.870119    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.870214    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.870319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.870252    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.971795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.971735    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.971893    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8c7w\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.971928    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.971991    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.972028    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.972140    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.972173    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.972232    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.972377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.972261    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.973358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.973294    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.973358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.973304    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.973829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.973765    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.973946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.973842    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.974020 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.973955    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.975306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.975285    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:45.975659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:45.975630    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:46.034067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.033994    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:46.034376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.034352    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8c7w\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:46.061209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.061165    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:46.511504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.511477    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"]
Oct 30 20:02:46.512999 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:46.512968    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda9fa23_1d23_4ea8_b94b_ce4c58ce7b27.slice/crio-3c742e5012976481a58222855e345534ab8640f59d5a6b50d0136164afda61be WatchSource:0}: Error finding container 3c742e5012976481a58222855e345534ab8640f59d5a6b50d0136164afda61be: Status 404 returned error can't find the container with id 3c742e5012976481a58222855e345534ab8640f59d5a6b50d0136164afda61be
Oct 30 20:02:46.515706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.515670    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:46.515801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.515753    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:46.515851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.515818    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:46.557920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.557891    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"]
Oct 30 20:02:46.562212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.562187    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.619635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.618611    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:46.619635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.618618    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:46.619635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.618730    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:46.619635 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.618895    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:46.624273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.624237    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"]
Oct 30 20:02:46.679865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.679832    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.680053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.679974    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.680123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.680057    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hhf\" (UniqueName: \"kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.780937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.780846    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.781378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.780939    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.781378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.780996    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hhf\" (UniqueName: \"kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.781626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.781606    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.783332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.783311    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.843049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.843013    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hhf\" (UniqueName: \"kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf\") pod \"maas-api-revoke-sub-7d10a8f0-ltx5c\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:46.876878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:46.876843    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:47.269813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:47.264127    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"]
Oct 30 20:02:47.271900 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:47.271869    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf63ed7a_4335_4e6e_952c_75fa436da582.slice/crio-4b9f2a164ffb130586f53953856694877519fbec2841f97c0951402d5c490a49 WatchSource:0}: Error finding container 4b9f2a164ffb130586f53953856694877519fbec2841f97c0951402d5c490a49: Status 404 returned error can't find the container with id 4b9f2a164ffb130586f53953856694877519fbec2841f97c0951402d5c490a49
Oct 30 20:02:47.280265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:47.280235    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" event={"ID":"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27","Type":"ContainerStarted","Data":"35c8d37fb7ce36612f5f84189a3ef31b61d864de14cf6d9592f6bacf5bfd8872"}
Oct 30 20:02:47.280385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:47.280296    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" event={"ID":"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27","Type":"ContainerStarted","Data":"3c742e5012976481a58222855e345534ab8640f59d5a6b50d0136164afda61be"}
Oct 30 20:02:47.286494 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:47.286462    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c" event={"ID":"cf63ed7a-4335-4e6e-952c-75fa436da582","Type":"ContainerStarted","Data":"4b9f2a164ffb130586f53953856694877519fbec2841f97c0951402d5c490a49"}
Oct 30 20:02:47.413324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:47.413275    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podStartSLOduration=2.413260275 podStartE2EDuration="2.413260275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:47.412550533 +0000 UTC m=+1263.576952187" watchObservedRunningTime="2026-10-30 20:02:47.413260275 +0000 UTC m=+1263.577661917"
Oct 30 20:02:48.066045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:48.066003    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:48.067642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:48.067616    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:02:48.067781 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:48.067662    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:02:48.300966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:48.300927    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c" event={"ID":"cf63ed7a-4335-4e6e-952c-75fa436da582","Type":"ContainerStarted","Data":"c0bc262f6c40210bd0d431e6b48c978c48673f629e02fae83513f5ac4b9b472c"}
Oct 30 20:02:48.429353 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:48.429298    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c" podStartSLOduration=2.429285239 podStartE2EDuration="2.429285239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:48.42647155 +0000 UTC m=+1264.590873193" watchObservedRunningTime="2026-10-30 20:02:48.429285239 +0000 UTC m=+1264.593686880"
Oct 30 20:02:49.067190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:49.067150    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:02:49.067604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:49.067213    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:02:49.308259 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:49.308215    4917 generic.go:356] "Generic (PLEG): container finished" podID="cf63ed7a-4335-4e6e-952c-75fa436da582" containerID="c0bc262f6c40210bd0d431e6b48c978c48673f629e02fae83513f5ac4b9b472c" exitCode=0
Oct 30 20:02:49.308468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:49.308299    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c" event={"ID":"cf63ed7a-4335-4e6e-952c-75fa436da582","Type":"ContainerDied","Data":"c0bc262f6c40210bd0d431e6b48c978c48673f629e02fae83513f5ac4b9b472c"}
Oct 30 20:02:49.700468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:49.700435    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:49.713382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:49.713353    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:50.062854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.062174    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:02:50.062854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.062249    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:02:50.067305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.067274    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"]
Oct 30 20:02:50.071153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.071130    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.172482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.172452    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"]
Oct 30 20:02:50.216081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216049    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216087    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216106    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216169    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216270    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216289    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216378    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216441    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.216673 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.216528    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz97r\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317294    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz97r\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317384    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317417    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317443    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317477    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317534    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317558    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317586    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317632    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.317795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.317762    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.318473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.318411    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.318473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.318433    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.318656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.318490    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.318656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.318586    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.320122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.320099    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.320478 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.320458    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.326366 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.326342    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:02:50.379814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.379763    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz97r\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.379959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.379860    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8rll6\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.384756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.384728    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:50.511989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.511960    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:50.620861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.620778    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca\") pod \"cf63ed7a-4335-4e6e-952c-75fa436da582\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") "
Oct 30 20:02:50.621030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.620900    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca\") pod \"cf63ed7a-4335-4e6e-952c-75fa436da582\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") "
Oct 30 20:02:50.621030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.621008    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf\" (UniqueName: \"kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf\") pod \"cf63ed7a-4335-4e6e-952c-75fa436da582\" (UID: \"cf63ed7a-4335-4e6e-952c-75fa436da582\") "
Oct 30 20:02:50.621298 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.621271    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca" pod "cf63ed7a-4335-4e6e-952c-75fa436da582" (UID: "cf63ed7a-4335-4e6e-952c-75fa436da582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:50.621405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.621392    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:50.622730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.622707    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca" pod "cf63ed7a-4335-4e6e-952c-75fa436da582" (UID: "cf63ed7a-4335-4e6e-952c-75fa436da582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:50.623238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.623221    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf" pod "cf63ed7a-4335-4e6e-952c-75fa436da582" (UID: "cf63ed7a-4335-4e6e-952c-75fa436da582"). InnerVolumeSpecName "kube-api-access-f2hhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:50.722972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.722928    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2hhf\" (UniqueName: \"kubernetes.io/projected/cf63ed7a-4335-4e6e-952c-75fa436da582-kube-api-access-f2hhf\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:50.722972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.722969    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf63ed7a-4335-4e6e-952c-75fa436da582-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:50.878274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.878237    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"]
Oct 30 20:02:50.879204 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:50.879172    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484556cc_6624_446f_a8c8_949ab75c31e2.slice/crio-28b87120d8fe5e6f460193ff5fc66ff18cdc815fc0e04f80eb39c5464ca7a436 WatchSource:0}: Error finding container 28b87120d8fe5e6f460193ff5fc66ff18cdc815fc0e04f80eb39c5464ca7a436: Status 404 returned error can't find the container with id 28b87120d8fe5e6f460193ff5fc66ff18cdc815fc0e04f80eb39c5464ca7a436
Oct 30 20:02:50.881685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.881654    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:50.881759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.881726    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:50.881826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:50.881759    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:02:51.063796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.063207    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:02:51.063796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.063278    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:02:51.322472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.322443    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"
Oct 30 20:02:51.322927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.322443    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c" event={"ID":"cf63ed7a-4335-4e6e-952c-75fa436da582","Type":"ContainerDied","Data":"4b9f2a164ffb130586f53953856694877519fbec2841f97c0951402d5c490a49"}
Oct 30 20:02:51.322927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.322567    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b9f2a164ffb130586f53953856694877519fbec2841f97c0951402d5c490a49"
Oct 30 20:02:51.324573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.324544    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" event={"ID":"484556cc-6624-446f-a8c8-949ab75c31e2","Type":"ContainerStarted","Data":"5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa"}
Oct 30 20:02:51.324720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.324599    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" event={"ID":"484556cc-6624-446f-a8c8-949ab75c31e2","Type":"ContainerStarted","Data":"28b87120d8fe5e6f460193ff5fc66ff18cdc815fc0e04f80eb39c5464ca7a436"}
Oct 30 20:02:51.385852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.385807    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:51.387375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.387350    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:51.387439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.387402    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:51.472966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:51.472913    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podStartSLOduration=2.47289851 podStartE2EDuration="2.47289851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:51.467097398 +0000 UTC m=+1267.631499041" watchObservedRunningTime="2026-10-30 20:02:51.47289851 +0000 UTC m=+1267.637300152"
Oct 30 20:02:52.070796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:52.066910    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:02:52.070796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:52.066982    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:02:52.385293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:52.385191    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:52.385893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:52.385860    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:53.062146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:53.062108    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:02:53.063107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:53.063075    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:02:53.385363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:53.385268    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:53.385363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:53.385322    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:54.070319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:54.070047    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:54.070727 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:54.070706    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:54.071613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:54.071596    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:02:54.385707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:54.385621    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:54.385707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:54.385679    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:55.385403 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:55.385372    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:55.385559 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:55.385421    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:56.386007 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:56.385976    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:56.386373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:56.386025    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:57.251473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.251436    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"]
Oct 30 20:02:57.251997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.251983    4917 cpu_manager.go:414] "containerMap: removing container" podUID="cf63ed7a-4335-4e6e-952c-75fa436da582" containerName="revoke-keys"
Oct 30 20:02:57.252046 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.251999    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf63ed7a-4335-4e6e-952c-75fa436da582" containerName="revoke-keys"
Oct 30 20:02:57.252109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.252100    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf63ed7a-4335-4e6e-952c-75fa436da582" containerName="revoke-keys"
Oct 30 20:02:57.255684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.255666    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.309566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.309524    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpl5b\" (UniqueName: \"kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.309725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.309587    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.309814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.309758    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.353204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.353172    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"]
Oct 30 20:02:57.362534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.362507    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:57.362679 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.362663    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:57.362728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.362663    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:57.362913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.362893    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:57.385903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.385869    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:57.386048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.385929    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:57.411039 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.410995    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpl5b\" (UniqueName: \"kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.411190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.411086    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.411238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.411205    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.411730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.411711    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.413849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.413825    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.474663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.474623    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpl5b\" (UniqueName: \"kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b\") pod \"maas-api-revoke-sub-17599f98-t4nsz\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.569838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.569707    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:02:57.971076 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:02:57.971030    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a5512d_1d26_4a2d_af4f_b55c0483a619.slice/crio-f32dd35407d27d5903a48f62a5719d091b670445653d049c8003efb20dc256e4 WatchSource:0}: Error finding container f32dd35407d27d5903a48f62a5719d091b670445653d049c8003efb20dc256e4: Status 404 returned error can't find the container with id f32dd35407d27d5903a48f62a5719d091b670445653d049c8003efb20dc256e4
Oct 30 20:02:57.972978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:57.972945    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"]
Oct 30 20:02:58.385533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:58.385498    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8rll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:02:58.385687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:58.385550    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:02:58.398574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:58.398529    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz" event={"ID":"91a5512d-1d26-4a2d-af4f-b55c0483a619","Type":"ContainerStarted","Data":"64b9f1d08727addd32f5b0d20688ae5e2ce3d32ce0bda185a5ee37a39bb97093"}
Oct 30 20:02:58.399074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:58.398584    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz" event={"ID":"91a5512d-1d26-4a2d-af4f-b55c0483a619","Type":"ContainerStarted","Data":"f32dd35407d27d5903a48f62a5719d091b670445653d049c8003efb20dc256e4"}
Oct 30 20:02:58.519609 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:58.519500    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz" podStartSLOduration=1.519485111 podStartE2EDuration="1.519485111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:58.516789107 +0000 UTC m=+1274.681190750" watchObservedRunningTime="2026-10-30 20:02:58.519485111 +0000 UTC m=+1274.683886750"
Oct 30 20:02:59.409065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:59.409024    4917 generic.go:356] "Generic (PLEG): container finished" podID="91a5512d-1d26-4a2d-af4f-b55c0483a619" containerID="64b9f1d08727addd32f5b0d20688ae5e2ce3d32ce0bda185a5ee37a39bb97093" exitCode=0
Oct 30 20:02:59.409487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:59.409110    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz" event={"ID":"91a5512d-1d26-4a2d-af4f-b55c0483a619","Type":"ContainerDied","Data":"64b9f1d08727addd32f5b0d20688ae5e2ce3d32ce0bda185a5ee37a39bb97093"}
Oct 30 20:02:59.467072 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:59.466986    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:59.468104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:59.468080    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:02:59.468899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:02:59.468871    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:03:00.570171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.570142    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:03:00.640977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.640939    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca\") pod \"91a5512d-1d26-4a2d-af4f-b55c0483a619\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") "
Oct 30 20:03:00.641157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.641029    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b\" (UniqueName: \"kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b\") pod \"91a5512d-1d26-4a2d-af4f-b55c0483a619\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") "
Oct 30 20:03:00.641157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.641141    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca\") pod \"91a5512d-1d26-4a2d-af4f-b55c0483a619\" (UID: \"91a5512d-1d26-4a2d-af4f-b55c0483a619\") "
Oct 30 20:03:00.641498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.641464    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca" pod "91a5512d-1d26-4a2d-af4f-b55c0483a619" (UID: "91a5512d-1d26-4a2d-af4f-b55c0483a619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:00.643056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.643035    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca" pod "91a5512d-1d26-4a2d-af4f-b55c0483a619" (UID: "91a5512d-1d26-4a2d-af4f-b55c0483a619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:00.643226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.643201    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b" pod "91a5512d-1d26-4a2d-af4f-b55c0483a619" (UID: "91a5512d-1d26-4a2d-af4f-b55c0483a619"). InnerVolumeSpecName "kube-api-access-hpl5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:00.742892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.742853    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:03:00.742892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.742888    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a5512d-1d26-4a2d-af4f-b55c0483a619-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:03:00.742892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:00.742899    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpl5b\" (UniqueName: \"kubernetes.io/projected/91a5512d-1d26-4a2d-af4f-b55c0483a619-kube-api-access-hpl5b\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:03:01.422980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:01.422935    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz" event={"ID":"91a5512d-1d26-4a2d-af4f-b55c0483a619","Type":"ContainerDied","Data":"f32dd35407d27d5903a48f62a5719d091b670445653d049c8003efb20dc256e4"}
Oct 30 20:03:01.423261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:01.423007    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f32dd35407d27d5903a48f62a5719d091b670445653d049c8003efb20dc256e4"
Oct 30 20:03:01.423261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:01.422956    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"
Oct 30 20:03:09.680595 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:03:09.680546    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:03:16.279674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.279629    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"]
Oct 30 20:03:16.280167 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.280152    4917 cpu_manager.go:414] "containerMap: removing container" podUID="91a5512d-1d26-4a2d-af4f-b55c0483a619" containerName="revoke-keys"
Oct 30 20:03:16.280213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.280168    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a5512d-1d26-4a2d-af4f-b55c0483a619" containerName="revoke-keys"
Oct 30 20:03:16.280292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.280283    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a5512d-1d26-4a2d-af4f-b55c0483a619" containerName="revoke-keys"
Oct 30 20:03:16.284646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.284627    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.352555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.352521    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bebb42-openshift-default-dockercfg-frbd5\""
Oct 30 20:03:16.405245 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.405211    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"]
Oct 30 20:03:16.414636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414603    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.414789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414675    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.414789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414741    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr56h\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.414884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414856    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.414919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414887    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.414979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414964    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.415014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.414987    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.415014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.415009    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.415113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.415078    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.516568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516530    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.516751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516578    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr56h\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.516751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516637    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.516751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516673    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.516751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516730    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516756    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516809    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516863    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516938    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.516963    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.517053    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.517119    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.517201    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.517675 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.517654    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.519524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.519502    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.519652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.519634    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.580565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.580482    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.580716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.580608    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr56h\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:16.598434 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:16.598401    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:17.029817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.029787    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"]
Oct 30 20:03:17.030208 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:03:17.030176    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ebf533_bdbc_4115_b099_69f34261409c.slice/crio-de5ec6441b867f5962722cb46414b15965bf69ed57279806b0a7214f0e73c23e WatchSource:0}: Error finding container de5ec6441b867f5962722cb46414b15965bf69ed57279806b0a7214f0e73c23e: Status 404 returned error can't find the container with id de5ec6441b867f5962722cb46414b15965bf69ed57279806b0a7214f0e73c23e
Oct 30 20:03:17.032832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.032794    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:17.032914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.032881    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:17.032962 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.032918    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:17.539374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.539334    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" event={"ID":"a8ebf533-bdbc-4115-b099-69f34261409c","Type":"ContainerStarted","Data":"a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c"}
Oct 30 20:03:17.539826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.539430    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" event={"ID":"a8ebf533-bdbc-4115-b099-69f34261409c","Type":"ContainerStarted","Data":"de5ec6441b867f5962722cb46414b15965bf69ed57279806b0a7214f0e73c23e"}
Oct 30 20:03:17.598838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.598800    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:17.668344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:17.668286    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podStartSLOduration=1.668271736 podStartE2EDuration="1.668271736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:17.666180926 +0000 UTC m=+1293.830582570" watchObservedRunningTime="2026-10-30 20:03:17.668271736 +0000 UTC m=+1293.832673379"
Oct 30 20:03:18.600904 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:18.600867    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:03:18.601302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:18.600931    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:03:19.599642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:19.599603    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:03:19.599849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:19.599673    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:03:20.598811 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:20.598749    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:03:20.599310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:20.598849    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:03:21.599510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:21.599473    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:03:21.599968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:21.599544    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:03:22.599336 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:22.599298    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:03:22.599548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:22.599359    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:03:23.698347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:23.698311    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:23.699083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:23.699057    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:23.699720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:23.699703    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:03:23.820338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:23.820242    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:03:28.939272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.939239    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"]
Oct 30 20:03:28.944651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.944619    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:28.972944 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.972901    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:28.973138 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.972961    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx79l\" (UniqueName: \"kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:28.973218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.973165    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:28.973323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.973303    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:28.973395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.973336    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:28.973395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:28.973359    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.049063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.049036    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"openshift-service-ca.crt\""
Oct 30 20:03:29.049392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.049373    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"kube-root-ca.crt\""
Oct 30 20:03:29.049942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.049913    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"default-dockercfg-h8x5s\""
Oct 30 20:03:29.050038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.049966    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"distinct-w4-bebb42-kserve-self-signed-certs\""
Oct 30 20:03:29.053051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.053030    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"]
Oct 30 20:03:29.074274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074249    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074373    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074400    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074422    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074457    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074487    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx79l\" (UniqueName: \"kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074900    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.074924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074915    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.075065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.074993    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.076931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.076909    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.077019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.076919    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.136740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.136705    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx79l\" (UniqueName: \"kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l\") pod \"distinct-w4-bebb42-kserve-7995c4cff8-qp5nd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.266604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.266517    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:29.682884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.682857    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"]
Oct 30 20:03:29.683686 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:03:29.683664    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedbeb7ac_aabc_4335_803a_def6b3eaa5bd.slice/crio-32d268f04839fd418bdbaceb124550ac8646bcd485e7e5efc123626e9526a3f2 WatchSource:0}: Error finding container 32d268f04839fd418bdbaceb124550ac8646bcd485e7e5efc123626e9526a3f2: Status 404 returned error can't find the container with id 32d268f04839fd418bdbaceb124550ac8646bcd485e7e5efc123626e9526a3f2
Oct 30 20:03:29.756403 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.756368    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"]
Oct 30 20:03:29.760826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.760804    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782339    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782397    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782460    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782493    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782535    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782560    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782621    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j785\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782646    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.782752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.782669    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.814950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.814917    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f75272-openshift-default-dockercfg-9blkz\""
Oct 30 20:03:29.870438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.870403    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"]
Oct 30 20:03:29.883428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883396    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883438    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883464    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883617    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883690    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883736    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883843    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j785\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.883897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883881    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.884002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883910    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.884002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.883931    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.884413 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.884099    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.884413 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.884283    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.884647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.884567    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.884741 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.884722    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.886262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.886232    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.886838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.886813    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.947827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.947716    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:29.948205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:29.947962    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j785\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-4wsck\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:30.076913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.076880    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:30.514538 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.514503    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"]
Oct 30 20:03:30.514968 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:03:30.514940    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1abb2b7_0c76_49f0_8f81_3686caf389cd.slice/crio-79d905423e58f596185e94ac6f763637078ae0643f9d6f353ec88ca3e247bbc9 WatchSource:0}: Error finding container 79d905423e58f596185e94ac6f763637078ae0643f9d6f353ec88ca3e247bbc9: Status 404 returned error can't find the container with id 79d905423e58f596185e94ac6f763637078ae0643f9d6f353ec88ca3e247bbc9
Oct 30 20:03:30.517914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.517877    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:30.518016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.517958    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:30.518016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.518001    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:30.649817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.649749    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" event={"ID":"f1abb2b7-0c76-49f0-8f81-3686caf389cd","Type":"ContainerStarted","Data":"13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481"}
Oct 30 20:03:30.649984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.649843    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" event={"ID":"f1abb2b7-0c76-49f0-8f81-3686caf389cd","Type":"ContainerStarted","Data":"79d905423e58f596185e94ac6f763637078ae0643f9d6f353ec88ca3e247bbc9"}
Oct 30 20:03:30.651550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.651521    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" event={"ID":"edbeb7ac-aabc-4335-803a-def6b3eaa5bd","Type":"ContainerStarted","Data":"08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6"}
Oct 30 20:03:30.651685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.651561    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" event={"ID":"edbeb7ac-aabc-4335-803a-def6b3eaa5bd","Type":"ContainerStarted","Data":"32d268f04839fd418bdbaceb124550ac8646bcd485e7e5efc123626e9526a3f2"}
Oct 30 20:03:30.779672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.779557    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podStartSLOduration=1.77954221 podStartE2EDuration="1.77954221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:30.778849798 +0000 UTC m=+1306.943251441" watchObservedRunningTime="2026-10-30 20:03:30.77954221 +0000 UTC m=+1306.943943852"
Oct 30 20:03:30.899147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:30.899087    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podStartSLOduration=2.8990730300000003 podStartE2EDuration="2.89907303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:30.897119503 +0000 UTC m=+1307.061521188" watchObservedRunningTime="2026-10-30 20:03:30.89907303 +0000 UTC m=+1307.063474671"
Oct 30 20:03:31.077149 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:31.077055    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:31.078544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:31.078520    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:03:31.078670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:31.078589    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:03:31.267671 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:31.267624    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:32.077861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:32.077828    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:03:32.078243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:32.077881    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:03:33.077603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:33.077567    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:03:33.077889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:33.077623    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:03:34.077506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:34.077464    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:03:34.077702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:34.077517    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:03:35.077476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:35.077437    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:03:35.077957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:35.077507    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:03:36.077705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:36.077665    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:03:36.078137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:36.077731    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:03:37.215206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:37.215166    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:37.216044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:37.216023    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:37.216887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:37.216866    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:03:39.267523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:39.267481    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:39.294406 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:39.294375    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:39.683219 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:03:39.683177    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:03:39.743704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:39.743672    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:03:45.956479 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:45.956443    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"]
Oct 30 20:03:45.960390 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:45.960370    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.013756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.013726    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-779921-openshift-default-dockercfg-rs78c\""
Oct 30 20:03:46.067698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.067667    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"]
Oct 30 20:03:46.085233 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085201    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085399 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085257    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085408    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmlc\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085497    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085562    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085625    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085684    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085843    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.085918 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.085895    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187178    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187227    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187275    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187308    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmlc\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187346    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187397    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187448    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187481    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187549    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187755    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187858    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.187897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.187890    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.188140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.188118    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.188280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.188263    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.189623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.189604    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.189980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.189960    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.249624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.249539    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.249792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.249690    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmlc\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.273709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.273683    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:46.709714 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.709685    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"]
Oct 30 20:03:46.710067 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:03:46.710041    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6240e660_a35c_4f25_8831_9156367ae6ad.slice/crio-5bec3d187d06e8a8feb556ab433d1fcf4afa88d4904d4162bd06de82855fe07f WatchSource:0}: Error finding container 5bec3d187d06e8a8feb556ab433d1fcf4afa88d4904d4162bd06de82855fe07f: Status 404 returned error can't find the container with id 5bec3d187d06e8a8feb556ab433d1fcf4afa88d4904d4162bd06de82855fe07f
Oct 30 20:03:46.712391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.712358    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:46.712489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.712426    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:46.712530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.712474    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:03:46.784157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.784124    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" event={"ID":"6240e660-a35c-4f25-8831-9156367ae6ad","Type":"ContainerStarted","Data":"0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22"}
Oct 30 20:03:46.784312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.784179    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" event={"ID":"6240e660-a35c-4f25-8831-9156367ae6ad","Type":"ContainerStarted","Data":"5bec3d187d06e8a8feb556ab433d1fcf4afa88d4904d4162bd06de82855fe07f"}
Oct 30 20:03:46.911927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:46.911861    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podStartSLOduration=1.911842555 podStartE2EDuration="1.911842555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:46.90579912 +0000 UTC m=+1323.070200762" watchObservedRunningTime="2026-10-30 20:03:46.911842555 +0000 UTC m=+1323.076244197"
Oct 30 20:03:47.273986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:47.273945    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:47.275673 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:47.275636    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:47.275818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:47.275692    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:48.274315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:48.274275    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:48.274729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:48.274332    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:49.274297 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:49.274256    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:49.274492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:49.274321    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:50.274879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:50.274844    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:50.275279 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:50.274902    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:51.274546 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:51.274500    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:51.274764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:51.274569    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:52.274373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:52.274321    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:52.274819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:52.274417    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:53.274193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:53.274148    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:03:53.274372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:53.274238    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:03:54.279080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:54.279047    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:54.279461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:54.279406    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:54.280092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:54.280071    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:03:56.503381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:03:56.503347    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:04:09.680646 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:04:09.680602    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:04:12.140142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.140106    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"]
Oct 30 20:04:12.144306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.144288    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.204977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.204945    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"]
Oct 30 20:04:12.217167 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.217135    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.217323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.217171    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.217409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.217381    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcfv\" (UniqueName: \"kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.250246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.250211    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:12.250414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.250225    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:12.250414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.250266    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:12.250414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.250269    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:04:12.318844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.318765    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcfv\" (UniqueName: \"kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.319049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.318948    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.319049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.318967    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.319749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.319724    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.321429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.321408    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.381292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.381263    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcfv\" (UniqueName: \"kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv\") pod \"maas-api-revoke-sub-fe0d1bd1-6x8l6\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.457911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.457874    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:12.845950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.845915    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"]
Oct 30 20:04:12.846970 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:04:12.846934    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c335b54_1675_4358_8cf9_44e7a07a7287.slice/crio-ca808c1c8ac261607e954d75cb3b2022632fa2ecf939883e4c2de32f689ba2fc WatchSource:0}: Error finding container ca808c1c8ac261607e954d75cb3b2022632fa2ecf939883e4c2de32f689ba2fc: Status 404 returned error can't find the container with id ca808c1c8ac261607e954d75cb3b2022632fa2ecf939883e4c2de32f689ba2fc
Oct 30 20:04:12.969763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:12.969722    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6" event={"ID":"1c335b54-1675-4358-8cf9-44e7a07a7287","Type":"ContainerStarted","Data":"ca808c1c8ac261607e954d75cb3b2022632fa2ecf939883e4c2de32f689ba2fc"}
Oct 30 20:04:13.977872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:13.977836    4917 generic.go:356] "Generic (PLEG): container finished" podID="1c335b54-1675-4358-8cf9-44e7a07a7287" containerID="b1b1cd73bc3c9df82e3e1d9ebed1123f23781fb991a1a4be1a392143c1fa4478" exitCode=0
Oct 30 20:04:13.978308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:13.977923    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6" event={"ID":"1c335b54-1675-4358-8cf9-44e7a07a7287","Type":"ContainerDied","Data":"b1b1cd73bc3c9df82e3e1d9ebed1123f23781fb991a1a4be1a392143c1fa4478"}
Oct 30 20:04:14.291730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.291679    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"]
Oct 30 20:04:14.298625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.298585    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.352713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.352676    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"]
Oct 30 20:04:14.442790 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.442745    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.442983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.442832    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvq6\" (UniqueName: \"kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.442983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.442873    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.544365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.544273    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.544365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.544330    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvq6\" (UniqueName: \"kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.544621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.544372    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.545141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.545117    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.546876 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.546852    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.609420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.609386    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvq6\" (UniqueName: \"kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6\") pod \"maas-api-revoke-sub-bed2a950-lkn42\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.612268 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.612238    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:14.997276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:14.997244    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"]
Oct 30 20:04:14.999103 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:04:14.999073    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf001b444_9423_49cb_ae7f_a9d33ed9bca4.slice/crio-beaabf9546db8b557f7ec48dce14359dfd2736fd353c636d649924c361481272 WatchSource:0}: Error finding container beaabf9546db8b557f7ec48dce14359dfd2736fd353c636d649924c361481272: Status 404 returned error can't find the container with id beaabf9546db8b557f7ec48dce14359dfd2736fd353c636d649924c361481272
Oct 30 20:04:15.125534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.125502    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:15.256191 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.256160    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca\") pod \"1c335b54-1675-4358-8cf9-44e7a07a7287\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") "
Oct 30 20:04:15.256325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.256236    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca\") pod \"1c335b54-1675-4358-8cf9-44e7a07a7287\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") "
Oct 30 20:04:15.256325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.256303    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv\" (UniqueName: \"kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv\") pod \"1c335b54-1675-4358-8cf9-44e7a07a7287\" (UID: \"1c335b54-1675-4358-8cf9-44e7a07a7287\") "
Oct 30 20:04:15.256616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.256584    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca" pod "1c335b54-1675-4358-8cf9-44e7a07a7287" (UID: "1c335b54-1675-4358-8cf9-44e7a07a7287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:15.256892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.256873    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:15.259047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.259023    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv" pod "1c335b54-1675-4358-8cf9-44e7a07a7287" (UID: "1c335b54-1675-4358-8cf9-44e7a07a7287"). InnerVolumeSpecName "kube-api-access-qzcfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:15.259147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.259042    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca" pod "1c335b54-1675-4358-8cf9-44e7a07a7287" (UID: "1c335b54-1675-4358-8cf9-44e7a07a7287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:15.357729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.357691    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c335b54-1675-4358-8cf9-44e7a07a7287-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:15.357729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.357726    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcfv\" (UniqueName: \"kubernetes.io/projected/1c335b54-1675-4358-8cf9-44e7a07a7287-kube-api-access-qzcfv\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:15.992521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.992488    4917 generic.go:356] "Generic (PLEG): container finished" podID="f001b444-9423-49cb-ae7f-a9d33ed9bca4" containerID="887b23b0c0b5ebebe62f76f5f75cf28d6fd755714e153376ca672ba253678ceb" exitCode=0
Oct 30 20:04:15.992716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.992577    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42" event={"ID":"f001b444-9423-49cb-ae7f-a9d33ed9bca4","Type":"ContainerDied","Data":"887b23b0c0b5ebebe62f76f5f75cf28d6fd755714e153376ca672ba253678ceb"}
Oct 30 20:04:15.992716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.992654    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42" event={"ID":"f001b444-9423-49cb-ae7f-a9d33ed9bca4","Type":"ContainerStarted","Data":"beaabf9546db8b557f7ec48dce14359dfd2736fd353c636d649924c361481272"}
Oct 30 20:04:15.993976 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.993957    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"
Oct 30 20:04:15.994104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.993994    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6" event={"ID":"1c335b54-1675-4358-8cf9-44e7a07a7287","Type":"ContainerDied","Data":"ca808c1c8ac261607e954d75cb3b2022632fa2ecf939883e4c2de32f689ba2fc"}
Oct 30 20:04:15.994104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:15.994031    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca808c1c8ac261607e954d75cb3b2022632fa2ecf939883e4c2de32f689ba2fc"
Oct 30 20:04:17.149350 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.149323    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:17.280213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.280123    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca\") pod \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") "
Oct 30 20:04:17.280391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.280216    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca\") pod \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") "
Oct 30 20:04:17.280452 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.280407    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6\" (UniqueName: \"kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6\") pod \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\" (UID: \"f001b444-9423-49cb-ae7f-a9d33ed9bca4\") "
Oct 30 20:04:17.280585 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.280556    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca" pod "f001b444-9423-49cb-ae7f-a9d33ed9bca4" (UID: "f001b444-9423-49cb-ae7f-a9d33ed9bca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:17.280945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.280913    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:17.282731 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.282699    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6" pod "f001b444-9423-49cb-ae7f-a9d33ed9bca4" (UID: "f001b444-9423-49cb-ae7f-a9d33ed9bca4"). InnerVolumeSpecName "kube-api-access-tmvq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:17.282855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.282818    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca" pod "f001b444-9423-49cb-ae7f-a9d33ed9bca4" (UID: "f001b444-9423-49cb-ae7f-a9d33ed9bca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:17.381861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.381825    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f001b444-9423-49cb-ae7f-a9d33ed9bca4-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:17.381861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:17.381855    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvq6\" (UniqueName: \"kubernetes.io/projected/f001b444-9423-49cb-ae7f-a9d33ed9bca4-kube-api-access-tmvq6\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:18.010319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:18.010283    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42" event={"ID":"f001b444-9423-49cb-ae7f-a9d33ed9bca4","Type":"ContainerDied","Data":"beaabf9546db8b557f7ec48dce14359dfd2736fd353c636d649924c361481272"}
Oct 30 20:04:18.010319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:18.010312    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"
Oct 30 20:04:18.010553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:18.010341    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beaabf9546db8b557f7ec48dce14359dfd2736fd353c636d649924c361481272"
Oct 30 20:04:25.019873 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.019835    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"]
Oct 30 20:04:25.020391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.020377    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f001b444-9423-49cb-ae7f-a9d33ed9bca4" containerName="revoke-keys"
Oct 30 20:04:25.020441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.020392    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f001b444-9423-49cb-ae7f-a9d33ed9bca4" containerName="revoke-keys"
Oct 30 20:04:25.020441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.020404    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1c335b54-1675-4358-8cf9-44e7a07a7287" containerName="revoke-keys"
Oct 30 20:04:25.020441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.020411    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c335b54-1675-4358-8cf9-44e7a07a7287" containerName="revoke-keys"
Oct 30 20:04:25.020544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.020505    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f001b444-9423-49cb-ae7f-a9d33ed9bca4" containerName="revoke-keys"
Oct 30 20:04:25.020544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.020515    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c335b54-1675-4358-8cf9-44e7a07a7287" containerName="revoke-keys"
Oct 30 20:04:25.024213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.024197    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.061698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.061668    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhm9r\" (UniqueName: \"kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.061882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.061756    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.061882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.061830    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.084625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.084595    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"]
Oct 30 20:04:25.130381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.130350    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:25.130381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.130380    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:25.130597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.130389    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:25.130892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.130868    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:04:25.163919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.163886    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhm9r\" (UniqueName: \"kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.164094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.163939    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.164145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.164111    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.164782 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.164752    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.166347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.166322    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.226697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.226660    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhm9r\" (UniqueName: \"kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r\") pod \"maas-api-revoke-sub-5aafd23a-4dqwp\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.338306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.338214    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:25.718456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:25.718426    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"]
Oct 30 20:04:25.719006 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:04:25.718972    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d899cdf_dd9d_4182_a6b6_405040315216.slice/crio-dfecd42a488057f0695da3b2fa5e78f8c686f092daafd0bc78efc3f53bf4586e WatchSource:0}: Error finding container dfecd42a488057f0695da3b2fa5e78f8c686f092daafd0bc78efc3f53bf4586e: Status 404 returned error can't find the container with id dfecd42a488057f0695da3b2fa5e78f8c686f092daafd0bc78efc3f53bf4586e
Oct 30 20:04:26.066982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:26.066946    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp" event={"ID":"5d899cdf-dd9d-4182-a6b6-405040315216","Type":"ContainerStarted","Data":"01700c11650e4dccd4f88b168801a2b9f881786eec2f05a3b4d4884364b9307a"}
Oct 30 20:04:26.067450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:26.067007    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp" event={"ID":"5d899cdf-dd9d-4182-a6b6-405040315216","Type":"ContainerStarted","Data":"dfecd42a488057f0695da3b2fa5e78f8c686f092daafd0bc78efc3f53bf4586e"}
Oct 30 20:04:26.189404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:26.189346    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp" podStartSLOduration=2.18933099 podStartE2EDuration="2.18933099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:26.187696961 +0000 UTC m=+1362.352098603" watchObservedRunningTime="2026-10-30 20:04:26.18933099 +0000 UTC m=+1362.353732631"
Oct 30 20:04:27.074913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:27.074877    4917 generic.go:356] "Generic (PLEG): container finished" podID="5d899cdf-dd9d-4182-a6b6-405040315216" containerID="01700c11650e4dccd4f88b168801a2b9f881786eec2f05a3b4d4884364b9307a" exitCode=0
Oct 30 20:04:27.075304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:27.074960    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp" event={"ID":"5d899cdf-dd9d-4182-a6b6-405040315216","Type":"ContainerDied","Data":"01700c11650e4dccd4f88b168801a2b9f881786eec2f05a3b4d4884364b9307a"}
Oct 30 20:04:28.223698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.223676    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:28.300136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.300107    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca\") pod \"5d899cdf-dd9d-4182-a6b6-405040315216\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") "
Oct 30 20:04:28.300327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.300149    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r\" (UniqueName: \"kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r\") pod \"5d899cdf-dd9d-4182-a6b6-405040315216\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") "
Oct 30 20:04:28.300327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.300245    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca\") pod \"5d899cdf-dd9d-4182-a6b6-405040315216\" (UID: \"5d899cdf-dd9d-4182-a6b6-405040315216\") "
Oct 30 20:04:28.300527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.300502    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca" pod "5d899cdf-dd9d-4182-a6b6-405040315216" (UID: "5d899cdf-dd9d-4182-a6b6-405040315216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:28.302346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.302321    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca" pod "5d899cdf-dd9d-4182-a6b6-405040315216" (UID: "5d899cdf-dd9d-4182-a6b6-405040315216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:28.302346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.302331    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r" pod "5d899cdf-dd9d-4182-a6b6-405040315216" (UID: "5d899cdf-dd9d-4182-a6b6-405040315216"). InnerVolumeSpecName "kube-api-access-jhm9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:28.401155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.401077    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:28.401155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.401105    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhm9r\" (UniqueName: \"kubernetes.io/projected/5d899cdf-dd9d-4182-a6b6-405040315216-kube-api-access-jhm9r\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:28.401155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:28.401115    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d899cdf-dd9d-4182-a6b6-405040315216-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:29.090448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:29.090412    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"
Oct 30 20:04:29.090448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:29.090428    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp" event={"ID":"5d899cdf-dd9d-4182-a6b6-405040315216","Type":"ContainerDied","Data":"dfecd42a488057f0695da3b2fa5e78f8c686f092daafd0bc78efc3f53bf4586e"}
Oct 30 20:04:29.090706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:29.090471    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfecd42a488057f0695da3b2fa5e78f8c686f092daafd0bc78efc3f53bf4586e"
Oct 30 20:04:33.063371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:04:33.063334    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:04:39.684219 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:04:39.684147    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:05:00.306214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.306136    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"]
Oct 30 20:05:00.306672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.306629    4917 cpu_manager.go:414] "containerMap: removing container" podUID="5d899cdf-dd9d-4182-a6b6-405040315216" containerName="revoke-keys"
Oct 30 20:05:00.306672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.306640    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d899cdf-dd9d-4182-a6b6-405040315216" containerName="revoke-keys"
Oct 30 20:05:00.306742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.306735    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d899cdf-dd9d-4182-a6b6-405040315216" containerName="revoke-keys"
Oct 30 20:05:00.311184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.311165    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391112    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391215    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391296    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391344    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391397    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391519 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391498    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391576 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391557    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqxp\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391695 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391679    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.391735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.391709    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.409653 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.409624    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"]
Oct 30 20:05:00.492404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492369    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492405    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492469    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492507    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492547    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492593    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492669    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492711    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.492895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492737    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqxp\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.493059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.492977    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.493059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.493032    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.493177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.493141    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.493232 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.493186    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.493443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.493406    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.495097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.495066    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.495281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.495263    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.554615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.554581    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.555004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.554981    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqxp\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:00.625882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:00.625802    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:01.073148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.073115    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"]
Oct 30 20:05:01.074334 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:05:01.074307    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4028bea6_3a70_4980_890f_4a86c7d44b2e.slice/crio-2c1e8902e32565ce25a3827b48521ff28eda248edc352fbded688f9c57246f27 WatchSource:0}: Error finding container 2c1e8902e32565ce25a3827b48521ff28eda248edc352fbded688f9c57246f27: Status 404 returned error can't find the container with id 2c1e8902e32565ce25a3827b48521ff28eda248edc352fbded688f9c57246f27
Oct 30 20:05:01.076926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.076893    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:01.076992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.076961    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:01.077027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.076990    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:01.139489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.139466    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"]
Oct 30 20:05:01.143303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.143286    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.197466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.197432    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f84d85-openshift-default-dockercfg-lv5m9\""
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202135    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202181    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7d7\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202215    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202299    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202393    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202440    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202463    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202572    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.206697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.202609    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.213963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.213929    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"]
Oct 30 20:05:01.303413 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303374    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303424    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303537    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7d7\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303585    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303673    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303796    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303954 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303837    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303954 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303859    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.303954 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.303882    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.304107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.304001    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.304174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.304148    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.304174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.304165    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.304376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.304353    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.304677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.304657    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.306093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.306067    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.306376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.306358    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.310320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.310290    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" event={"ID":"4028bea6-3a70-4980-890f-4a86c7d44b2e","Type":"ContainerStarted","Data":"89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6"}
Oct 30 20:05:01.310433 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.310341    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" event={"ID":"4028bea6-3a70-4980-890f-4a86c7d44b2e","Type":"ContainerStarted","Data":"2c1e8902e32565ce25a3827b48521ff28eda248edc352fbded688f9c57246f27"}
Oct 30 20:05:01.366384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.366293    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7d7\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.366384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.366292    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.433704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.433644    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podStartSLOduration=1.433629356 podStartE2EDuration="1.433629356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:01.433181258 +0000 UTC m=+1397.597582924" watchObservedRunningTime="2026-10-30 20:05:01.433629356 +0000 UTC m=+1397.598030999"
Oct 30 20:05:01.458384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.458345    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:01.573796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.569742    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"]
Oct 30 20:05:01.589800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.586073    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.626636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.626557    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:01.682973 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.682935    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"]
Oct 30 20:05:01.709257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709209    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709327    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709404    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709469    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709499    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jlq\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709578    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709746 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709609    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709746 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709732    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.709890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.709798    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.811604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811567    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.811818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811618    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.811818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811673    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.811818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811701    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.811818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811809    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811845    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811914    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811967    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.811996    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jlq\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.812175    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.812251    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.812446    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.812552    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.812983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.812957    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.814295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.814269    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.814640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.814620    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.838235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.838205    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"]
Oct 30 20:05:01.858114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.858082    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.874201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.874170    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"]
Oct 30 20:05:01.878478 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.878445    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.878618 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.878596    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jlq\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.879731 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.879704    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:01.879843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.879794    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:01.879897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.879839    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:01.914668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.906846    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.914755    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.914896    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl65c\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.914934    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.914965    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.915034    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.915062    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.915127    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.915169    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"]
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.915183    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:01.920468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:01.915222    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.018732    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.018868    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.018946    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.018999    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019054    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019137    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019225    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl65c\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019267    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019299    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019414    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019638    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.019763    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.020082    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.020900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.020108    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.024798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.023387    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.025394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.025371    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.083611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.083561    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.084217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.084188    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl65c\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.217932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.217898    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:02.330067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.329222    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" event={"ID":"68fa4fd4-1346-442b-9781-61233391c11b","Type":"ContainerStarted","Data":"2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d"}
Oct 30 20:05:02.330067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.329304    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" event={"ID":"68fa4fd4-1346-442b-9781-61233391c11b","Type":"ContainerStarted","Data":"8939a2854b650b1ae505fc6af33d02f0c900293e9b14462c28bef14787f0e667"}
Oct 30 20:05:02.338164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.338006    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"]
Oct 30 20:05:02.339815 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:05:02.339589    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadfae5c_c853_456d_bd09_e6f904b8a4b7.slice/crio-e5cc984b07265c0055c07e591ca1348e4e3140f765eb427e8e8c334a45dcdbb7 WatchSource:0}: Error finding container e5cc984b07265c0055c07e591ca1348e4e3140f765eb427e8e8c334a45dcdbb7: Status 404 returned error can't find the container with id e5cc984b07265c0055c07e591ca1348e4e3140f765eb427e8e8c334a45dcdbb7
Oct 30 20:05:02.378042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.361164    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:02.378042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.361253    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:02.378042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.361299    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:02.469957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.466009    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:02.476805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.476271    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podStartSLOduration=1.476249187 podStartE2EDuration="1.476249187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:02.467748718 +0000 UTC m=+1398.632150360" watchObservedRunningTime="2026-10-30 20:05:02.476249187 +0000 UTC m=+1398.640650831"
Oct 30 20:05:02.628797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.628745    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:05:02.629004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.628808    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:05:02.671654 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.671616    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:02.675859 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.675823    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:02.676088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.676063    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:02.676638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:02.674249    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"]
Oct 30 20:05:03.347802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.346518    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" event={"ID":"eadfae5c-c853-456d-bd09-e6f904b8a4b7","Type":"ContainerStarted","Data":"0851fdb4dcba61ab9950df67ae211331e5324c985e8ef16fed37bf6ae09e763d"}
Oct 30 20:05:03.347802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.346586    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" event={"ID":"eadfae5c-c853-456d-bd09-e6f904b8a4b7","Type":"ContainerStarted","Data":"e5cc984b07265c0055c07e591ca1348e4e3140f765eb427e8e8c334a45dcdbb7"}
Oct 30 20:05:03.378795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.376277    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" event={"ID":"621da606-4fd4-48d8-8bba-aacb03f1644f","Type":"ContainerStarted","Data":"fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600"}
Oct 30 20:05:03.378795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.376353    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" event={"ID":"621da606-4fd4-48d8-8bba-aacb03f1644f","Type":"ContainerStarted","Data":"3b5f030c3afcdfcd34f19b9e224a9d299df58eaec05e4f8c782544a4d10defd8"}
Oct 30 20:05:03.495476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.488939    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:05:03.495476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.489012    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:05:03.518798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.500545    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podStartSLOduration=2.5005227530000003 podStartE2EDuration="2.500522753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:03.489163266 +0000 UTC m=+1399.653564909" watchObservedRunningTime="2026-10-30 20:05:03.500522753 +0000 UTC m=+1399.664924396"
Oct 30 20:05:03.627857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.626620    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:05:03.627857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.626682    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:05:03.638414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.638018    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podStartSLOduration=2.637999976 podStartE2EDuration="2.637999976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:03.633490745 +0000 UTC m=+1399.797892389" watchObservedRunningTime="2026-10-30 20:05:03.637999976 +0000 UTC m=+1399.802401663"
Oct 30 20:05:03.926851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.925861    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:03.938796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.936929    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:03.938796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:03.936995    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:04.219794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.219154    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:04.240797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.226878    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:04.240797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.226940    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:04.470884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.470754    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:05:04.471480 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.471449    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:05:04.671178 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.671146    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:05:04.671461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.671427    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:05:04.754083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.753984    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:04.768947 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.768880    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:04.911686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.911645    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:04.911996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.911967    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:04.946939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:04.946889    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:05.220795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.219827    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:05.220795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.219921    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:05.317384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.317340    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:05.469797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.465296    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:05:05.469797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.465359    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:05:05.471059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.470976    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:05:05.632885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.632297    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:05:05.632885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.632388    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:05:05.668986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.668923    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:05.695951 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.695887    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:05:05.754966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.754860    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:05.755843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.755817    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:05:05.778911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.778862    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:05:05.779611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.779585    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:05:05.792913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:05.792867    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.050205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.050093    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.095833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.095793    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.137608 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.137541    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:06.137820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.102092    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:05:06.172799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.162731    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.323868 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.322185    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.323868 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.322299    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:06.543523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.543478    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.545190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.545157    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.552794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.546884    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.552794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.546931    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:05:06.553966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.553934    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.631365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.631268    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.631647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.631606    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:05:06.676795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.660623    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:05:06.676795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.660681    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:05:06.677057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.676939    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.677973 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.677308    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:05:06.691525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.691479    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:05:06.704879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.703011    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.704879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.703504    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:05:06.767794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.755869    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.858835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.856427    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:06.918097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.915022    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:05:06.927301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:06.921201    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.073271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.073219    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:05:07.097802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.073285    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:07.098696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.093097    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.180855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.177455    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.180855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.177972    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:05:07.269087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.268690    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.269087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.268753    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:07.496105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.495907    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.496105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.495972    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:05:07.571796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.567089    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.604795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.590414    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.647796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.641928    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.647796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.641997    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:05:07.821968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.821859    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.872400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.823919    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:05:07.889796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.823959    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.889796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.823980    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:05:07.889796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.866045    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:07.924226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.923900    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.924226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:07.923967    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:08.085878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.073610    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:05:08.220482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.220200    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.220482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.220266    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:08.226066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.224863    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:08.267543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.267110    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:08.500796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.485067    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.500796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.485139    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:05:08.589436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.589390    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:08.614253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.602712    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:08.704566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.703888    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:08.712370 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.705341    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:08.712370 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.706955    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:05:08.724125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.724095    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:05:08.725497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.725410    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:05:08.726501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.726478    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:05:08.735542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.735523    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:05:08.751088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.751014    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:05:08.828067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.828039    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:05:08.829087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.829063    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:05:08.831800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.831763    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:05:08.912854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.912757    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.913203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:08.913169    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:09.218872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.218837    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:09.219211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.219180    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:09.683803 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.683073    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:09.685071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.684952    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:09.686229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.686209    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:05:09.720250 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:05:09.720214    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:05:09.909268 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.909235    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:05:09.909446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:09.909292    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:05:10.177125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:10.177098    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:05:10.218990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:10.218953    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:05:10.219176 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:10.219020    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 20:05:10.932879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:10.932841    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:10.933387 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:10.933324    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:10.934992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:10.934968    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:05:11.223555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:11.223476    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:11.224002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:11.223977    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:11.224833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:11.224811    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:05:31.007371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.007339    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"]
Oct 30 20:05:31.012520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.012500    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.066862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.066821    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3ccc9f-openshift-default-dockercfg-sn7l4\""
Oct 30 20:05:31.075114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075076    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075233    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075276    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075314    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075334    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075376    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075411    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075644 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075464    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zhx\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.075644 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.075519    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.123061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.123026    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"]
Oct 30 20:05:31.177100 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177060    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177117    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zhx\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177157    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177191    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177255    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177300    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177353    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177382    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177537    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177624    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177662    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177847 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177766    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.177847 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.177815    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.178214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.178196    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.179684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.179654    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.179938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.179921    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.239555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.239520    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.239688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.239627    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zhx\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.326466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.326379    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:31.761194 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.761162    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"]
Oct 30 20:05:31.761449 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:05:31.761417    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926cf7cf_0c25_4581_9093_08bdeb409afa.slice/crio-12f7dc82d95227e65bee3224a44579118c4da7c11c7dbdcc8a628dc45ee74a79 WatchSource:0}: Error finding container 12f7dc82d95227e65bee3224a44579118c4da7c11c7dbdcc8a628dc45ee74a79: Status 404 returned error can't find the container with id 12f7dc82d95227e65bee3224a44579118c4da7c11c7dbdcc8a628dc45ee74a79
Oct 30 20:05:31.763794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.763749    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:31.763909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.763839    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:31.763909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:31.763884    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:05:32.138177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.138079    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" event={"ID":"926cf7cf-0c25-4581-9093-08bdeb409afa","Type":"ContainerStarted","Data":"b89d6a7a0221f8ff47a4c6a654354fbbb9314eb704784be31a47530ff5d1429a"}
Oct 30 20:05:32.138177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.138137    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" event={"ID":"926cf7cf-0c25-4581-9093-08bdeb409afa","Type":"ContainerStarted","Data":"12f7dc82d95227e65bee3224a44579118c4da7c11c7dbdcc8a628dc45ee74a79"}
Oct 30 20:05:32.268212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.265398    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podStartSLOduration=2.265378815 podStartE2EDuration="2.265378815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:32.258964398 +0000 UTC m=+1428.423366051" watchObservedRunningTime="2026-10-30 20:05:32.265378815 +0000 UTC m=+1428.429780458"
Oct 30 20:05:32.327213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.327174    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:32.328599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.328572    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:32.328732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.328636    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:32.922753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.922709    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"]
Oct 30 20:05:32.927109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.927085    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:32.981278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.981244    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:05:32.981507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.981249    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:05:32.981632 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.981282    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:05:32.981705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.981318    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:05:32.985932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:32.985903    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"]
Oct 30 20:05:33.004621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.004586    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.004829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.004785    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblpg\" (UniqueName: \"kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.004909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.004868    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.106043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.106004    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblpg\" (UniqueName: \"kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.106192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.106062    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.106258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.106228    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.106875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.106852    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.108693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.108669    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.169044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.169008    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblpg\" (UniqueName: \"kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg\") pod \"maas-api-revoke-sub-d4439202-sqvqp\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.248326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.246972    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:33.331804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.329753    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:33.331804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.329908    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:33.646817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:33.641402    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"]
Oct 30 20:05:34.178290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:34.178259    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp" event={"ID":"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba","Type":"ContainerStarted","Data":"ccb841b89668e710ab25003e85245dc302676aa8ea6f1c35308a522c54adbddd"}
Oct 30 20:05:34.326943 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:34.326908    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:34.327115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:34.326957    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:35.187081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:35.187046    4917 generic.go:356] "Generic (PLEG): container finished" podID="a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" containerID="b20476486ba9136c3273a31dfc8a4bfea2cdd1f9f14632a73dd3a7f3d8201e88" exitCode=0
Oct 30 20:05:35.187496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:35.187139    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp" event={"ID":"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba","Type":"ContainerDied","Data":"b20476486ba9136c3273a31dfc8a4bfea2cdd1f9f14632a73dd3a7f3d8201e88"}
Oct 30 20:05:35.326909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:35.326867    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:35.327085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:35.326925    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:36.326831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.326801    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:36.327158 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.326857    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:36.374093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.374062    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:36.472489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.472410    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca\") pod \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") "
Oct 30 20:05:36.472489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.472466    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg\" (UniqueName: \"kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg\") pod \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") "
Oct 30 20:05:36.472718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.472498    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca\") pod \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\" (UID: \"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba\") "
Oct 30 20:05:36.472822 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.472713    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca" pod "a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" (UID: "a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:36.472939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.472922    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:36.474677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.474653    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca" pod "a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" (UID: "a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:36.474802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.474682    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg" pod "a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" (UID: "a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba"). InnerVolumeSpecName "kube-api-access-zblpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:36.573678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.573639    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblpg\" (UniqueName: \"kubernetes.io/projected/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-kube-api-access-zblpg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:36.573678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:36.573676    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.204073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:37.204044    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"
Oct 30 20:05:37.204255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:37.204079    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp" event={"ID":"a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba","Type":"ContainerDied","Data":"ccb841b89668e710ab25003e85245dc302676aa8ea6f1c35308a522c54adbddd"}
Oct 30 20:05:37.204255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:37.204129    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb841b89668e710ab25003e85245dc302676aa8ea6f1c35308a522c54adbddd"
Oct 30 20:05:37.327749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:37.327712    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:37.328251 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:37.327813    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:38.327536 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:38.327496    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 20:05:38.327712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:38.327548    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 20:05:39.332082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.332024    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:39.332560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.332521    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:39.333482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.333461    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:05:39.686383 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:05:39.686315    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:05:39.932247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.932211    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"]
Oct 30 20:05:39.932811 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.932796    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" containerName="revoke-keys"
Oct 30 20:05:39.932864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.932812    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" containerName="revoke-keys"
Oct 30 20:05:39.932929 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.932920    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" containerName="revoke-keys"
Oct 30 20:05:39.937532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.937468    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:39.994144 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.994109    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"]
Oct 30 20:05:39.995541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:39.995513    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"distinct-w3-f84d85-kserve-self-signed-certs\""
Oct 30 20:05:40.015149 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.015117    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.015296 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.015271    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.015368 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.015337    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.015495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.015479    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.015571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.015539    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.015644 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.015629    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.116436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116400    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.116652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116478    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.116652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116530    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.116652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116602    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.116652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116635    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.116898 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116681    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.117012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.116956    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.117053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.117009    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.117108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.117086    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.119192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.119160    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.119446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.119425    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.179835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.179800    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb\") pod \"distinct-w3-f84d85-kserve-77895ddd7f-lp87v\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.251613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.251506    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:40.700663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:40.700624    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"]
Oct 30 20:05:40.701211 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:05:40.701189    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cc54869_c728_402e_bfd2_3c0caaf05a51.slice/crio-009ef69c8c281977fe19e03dd95a98f4338e82eb8f82ed384b66ebaf42d99fcb WatchSource:0}: Error finding container 009ef69c8c281977fe19e03dd95a98f4338e82eb8f82ed384b66ebaf42d99fcb: Status 404 returned error can't find the container with id 009ef69c8c281977fe19e03dd95a98f4338e82eb8f82ed384b66ebaf42d99fcb
Oct 30 20:05:41.235649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:41.235607    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" event={"ID":"0cc54869-c728-402e-bfd2-3c0caaf05a51","Type":"ContainerStarted","Data":"d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e"}
Oct 30 20:05:41.235882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:41.235661    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" event={"ID":"0cc54869-c728-402e-bfd2-3c0caaf05a51","Type":"ContainerStarted","Data":"009ef69c8c281977fe19e03dd95a98f4338e82eb8f82ed384b66ebaf42d99fcb"}
Oct 30 20:05:41.252373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:41.252342    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:41.356987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:41.356927    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podStartSLOduration=2.356914621 podStartE2EDuration="2.356914621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:41.35505233 +0000 UTC m=+1437.519453972" watchObservedRunningTime="2026-10-30 20:05:41.356914621 +0000 UTC m=+1437.521316262"
Oct 30 20:05:48.951396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:48.951362    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:05:50.252490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:50.252454    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:50.273075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:50.273042    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:50.316438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:50.316408    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:05:57.862564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.862525    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"]
Oct 30 20:05:57.867918 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.867901    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:57.929479 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.929446    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"]
Oct 30 20:05:57.972913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.972878    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:05:57.972913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.972896    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:05:57.972913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.972893    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:05:57.976453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:57.976427    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:05:58.008533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.008503    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k547v\" (UniqueName: \"kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.008691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.008618    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.008745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.008704    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.109982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.109950    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k547v\" (UniqueName: \"kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.110151 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.110082    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.110151 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.110129    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.111199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.111172    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.112728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.112671    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.177035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.176997    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k547v\" (UniqueName: \"kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v\") pod \"maas-api-revoke-sub-79af6e11-bzmwz\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.184647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.184612    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:05:58.570507 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:05:58.570478    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e30dbf2_2c4d_43ff_a585_676c520f40ee.slice/crio-923840557de60907ae2f89a334ec265722f777052bc558414e07125f7eb0f3d4 WatchSource:0}: Error finding container 923840557de60907ae2f89a334ec265722f777052bc558414e07125f7eb0f3d4: Status 404 returned error can't find the container with id 923840557de60907ae2f89a334ec265722f777052bc558414e07125f7eb0f3d4
Oct 30 20:05:58.571590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:58.571566    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"]
Oct 30 20:05:59.402210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:59.402180    4917 generic.go:356] "Generic (PLEG): container finished" podID="0e30dbf2-2c4d-43ff-a585-676c520f40ee" containerID="a704bb8007d803412c5d5814fcead2836e7b09d5b46942f0ed86e7cdf4a9fac4" exitCode=0
Oct 30 20:05:59.402581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:59.402256    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz" event={"ID":"0e30dbf2-2c4d-43ff-a585-676c520f40ee","Type":"ContainerDied","Data":"a704bb8007d803412c5d5814fcead2836e7b09d5b46942f0ed86e7cdf4a9fac4"}
Oct 30 20:05:59.402581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:05:59.402303    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz" event={"ID":"0e30dbf2-2c4d-43ff-a585-676c520f40ee","Type":"ContainerStarted","Data":"923840557de60907ae2f89a334ec265722f777052bc558414e07125f7eb0f3d4"}
Oct 30 20:06:00.569570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.569538    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:06:00.744371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.744333    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca\") pod \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") "
Oct 30 20:06:00.744581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.744402    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca\") pod \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") "
Oct 30 20:06:00.744581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.744441    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v\" (UniqueName: \"kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v\") pod \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\" (UID: \"0e30dbf2-2c4d-43ff-a585-676c520f40ee\") "
Oct 30 20:06:00.744826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.744791    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca" pod "0e30dbf2-2c4d-43ff-a585-676c520f40ee" (UID: "0e30dbf2-2c4d-43ff-a585-676c520f40ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:00.745111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.745085    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:00.746553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.746529    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca" pod "0e30dbf2-2c4d-43ff-a585-676c520f40ee" (UID: "0e30dbf2-2c4d-43ff-a585-676c520f40ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:00.746661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.746639    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v" pod "0e30dbf2-2c4d-43ff-a585-676c520f40ee" (UID: "0e30dbf2-2c4d-43ff-a585-676c520f40ee"). InnerVolumeSpecName "kube-api-access-k547v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:00.845974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.845942    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e30dbf2-2c4d-43ff-a585-676c520f40ee-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:00.845974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:00.845972    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k547v\" (UniqueName: \"kubernetes.io/projected/0e30dbf2-2c4d-43ff-a585-676c520f40ee-kube-api-access-k547v\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.418245 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:01.418213    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"
Oct 30 20:06:01.418424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:01.418250    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz" event={"ID":"0e30dbf2-2c4d-43ff-a585-676c520f40ee","Type":"ContainerDied","Data":"923840557de60907ae2f89a334ec265722f777052bc558414e07125f7eb0f3d4"}
Oct 30 20:06:01.418424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:01.418295    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="923840557de60907ae2f89a334ec265722f777052bc558414e07125f7eb0f3d4"
Oct 30 20:06:08.760217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.760183    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"]
Oct 30 20:06:08.761034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.761011    4917 cpu_manager.go:414] "containerMap: removing container" podUID="0e30dbf2-2c4d-43ff-a585-676c520f40ee" containerName="revoke-keys"
Oct 30 20:06:08.761148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.761035    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e30dbf2-2c4d-43ff-a585-676c520f40ee" containerName="revoke-keys"
Oct 30 20:06:08.761209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.761158    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e30dbf2-2c4d-43ff-a585-676c520f40ee" containerName="revoke-keys"
Oct 30 20:06:08.765015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.764995    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.835797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.833922    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"]
Oct 30 20:06:08.839464 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.839425    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ql99\" (UniqueName: \"kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.839607 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.839474    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.839668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.839619    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.877520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.877491    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:08.877520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.877505    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:08.877698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.877519    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:08.877698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.877605    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:08.940898 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.940859    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ql99\" (UniqueName: \"kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.941073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.940944    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.941073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.941020    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.941625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.941604    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:08.943660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:08.943632    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:09.004238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:09.004207    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ql99\" (UniqueName: \"kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99\") pod \"maas-api-revoke-sub-71263e87-kk978\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:09.079138 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:09.079037    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:09.472412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:09.472380    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"]
Oct 30 20:06:09.679649 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:06:09.679612    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:06:10.493235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:10.493187    4917 generic.go:356] "Generic (PLEG): container finished" podID="28dac310-e527-424b-8e3b-1cb4d3527872" containerID="ba034cfad8fa022fa6dae9cac4d244cf04e8a9b9539af3435e01422e59735f6b" exitCode=0
Oct 30 20:06:10.493642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:10.493270    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978" event={"ID":"28dac310-e527-424b-8e3b-1cb4d3527872","Type":"ContainerDied","Data":"ba034cfad8fa022fa6dae9cac4d244cf04e8a9b9539af3435e01422e59735f6b"}
Oct 30 20:06:10.493642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:10.493320    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978" event={"ID":"28dac310-e527-424b-8e3b-1cb4d3527872","Type":"ContainerStarted","Data":"b62fb50bf230cefcda0e602c2480cffa76f42b1e5d322351abf1f07aa211f5ea"}
Oct 30 20:06:11.084450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.084418    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"]
Oct 30 20:06:11.089481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.089455    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8042c72-95h6r"]
Oct 30 20:06:11.661082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.661059    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:11.775612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.775525    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca\") pod \"28dac310-e527-424b-8e3b-1cb4d3527872\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") "
Oct 30 20:06:11.775612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.775574    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca\") pod \"28dac310-e527-424b-8e3b-1cb4d3527872\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") "
Oct 30 20:06:11.775947 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.775797    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99\" (UniqueName: \"kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99\") pod \"28dac310-e527-424b-8e3b-1cb4d3527872\" (UID: \"28dac310-e527-424b-8e3b-1cb4d3527872\") "
Oct 30 20:06:11.775947 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.775891    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca" pod "28dac310-e527-424b-8e3b-1cb4d3527872" (UID: "28dac310-e527-424b-8e3b-1cb4d3527872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:11.776255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.776234    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:11.777690 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.777663    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca" pod "28dac310-e527-424b-8e3b-1cb4d3527872" (UID: "28dac310-e527-424b-8e3b-1cb4d3527872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:11.777832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.777813    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99" pod "28dac310-e527-424b-8e3b-1cb4d3527872" (UID: "28dac310-e527-424b-8e3b-1cb4d3527872"). InnerVolumeSpecName "kube-api-access-9ql99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:11.877801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.877743    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ql99\" (UniqueName: \"kubernetes.io/projected/28dac310-e527-424b-8e3b-1cb4d3527872-kube-api-access-9ql99\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:11.877801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:11.877796    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dac310-e527-424b-8e3b-1cb4d3527872-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:12.002726 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:12.002694    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a955af-47dd-4248-b353-9b3ba02e9f2f" path="/var/lib/kubelet/pods/c8a955af-47dd-4248-b353-9b3ba02e9f2f/volumes"
Oct 30 20:06:12.509354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:12.509324    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"
Oct 30 20:06:12.509354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:12.509341    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978" event={"ID":"28dac310-e527-424b-8e3b-1cb4d3527872","Type":"ContainerDied","Data":"b62fb50bf230cefcda0e602c2480cffa76f42b1e5d322351abf1f07aa211f5ea"}
Oct 30 20:06:12.509564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:12.509418    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b62fb50bf230cefcda0e602c2480cffa76f42b1e5d322351abf1f07aa211f5ea"
Oct 30 20:06:16.633833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.633790    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"]
Oct 30 20:06:16.634353 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.634339    4917 cpu_manager.go:414] "containerMap: removing container" podUID="28dac310-e527-424b-8e3b-1cb4d3527872" containerName="revoke-keys"
Oct 30 20:06:16.634398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.634354    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dac310-e527-424b-8e3b-1cb4d3527872" containerName="revoke-keys"
Oct 30 20:06:16.634490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.634481    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dac310-e527-424b-8e3b-1cb4d3527872" containerName="revoke-keys"
Oct 30 20:06:16.639184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.639167    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.696899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.696869    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"]
Oct 30 20:06:16.740572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.740537    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.740805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.740759    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.740897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.740885    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbnl\" (UniqueName: \"kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.741425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.741404    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:16.741543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.741421    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:16.741543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.741463    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:16.743626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.743607    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:16.842238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.842202    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.842409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.842347    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbnl\" (UniqueName: \"kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.842490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.842425    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.843312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.843043    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.845591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.845557    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.909254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.909186    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbnl\" (UniqueName: \"kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl\") pod \"maas-api-revoke-sub-7e323603-gdzs9\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:16.952132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:16.952100    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:17.193047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.193013    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"]
Oct 30 20:06:17.198160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.198135    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.247882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.247849    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cdfd\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.247902    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.247935    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.247959    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.248002    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.248069    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.248093    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.248149    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.248262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.248179    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.285019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.284982    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"]
Oct 30 20:06:17.342595 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:06:17.342565    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9c91d7_abb9_4731_a42c_2c7961a1de9a.slice/crio-9b8b6ac5574b6863126cd012f8d1ea4b1a465b2f434bec9af4d2a726f37cb8e9 WatchSource:0}: Error finding container 9b8b6ac5574b6863126cd012f8d1ea4b1a465b2f434bec9af4d2a726f37cb8e9: Status 404 returned error can't find the container with id 9b8b6ac5574b6863126cd012f8d1ea4b1a465b2f434bec9af4d2a726f37cb8e9
Oct 30 20:06:17.343915 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.343891    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"]
Oct 30 20:06:17.349444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349418    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349468    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349527    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349615    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349649    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349715    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349756    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349845    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cdfd\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349889    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.349999 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349888    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.350436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.350250    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.350436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.350329    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.350596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.349842    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.350909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.350888    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.352833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.352808    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.354969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.354857    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.418792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.418306    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cdfd\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.440899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.419396    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.513242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.513139    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:17.548801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:17.548737    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9" event={"ID":"1f9c91d7-abb9-4731-a42c-2c7961a1de9a","Type":"ContainerStarted","Data":"9b8b6ac5574b6863126cd012f8d1ea4b1a465b2f434bec9af4d2a726f37cb8e9"}
Oct 30 20:06:18.063979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.063936    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"]
Oct 30 20:06:18.117921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.113557    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:18.117921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.113644    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:18.117921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.113682    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:18.599800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.597352    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9" event={"ID":"1f9c91d7-abb9-4731-a42c-2c7961a1de9a","Type":"ContainerStarted","Data":"3508104b270cdab46c31bc097ad6928d19ff8e5b73044930de0a28f12f95b53e"}
Oct 30 20:06:18.604792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.600704    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" event={"ID":"36dc42f7-4cce-47b5-b6c2-c37577cce641","Type":"ContainerStarted","Data":"0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8"}
Oct 30 20:06:18.604792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.600759    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" event={"ID":"36dc42f7-4cce-47b5-b6c2-c37577cce641","Type":"ContainerStarted","Data":"851eb47ccbb1ad5ff4aaf5e10a13fc97215ba83ca9b1ad001773a38311224321"}
Oct 30 20:06:18.764011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.763946    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9" podStartSLOduration=2.763926691 podStartE2EDuration="2.763926691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:18.751585005 +0000 UTC m=+1474.915986648" watchObservedRunningTime="2026-10-30 20:06:18.763926691 +0000 UTC m=+1474.928328336"
Oct 30 20:06:18.886854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:18.877306    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podStartSLOduration=1.877287994 podStartE2EDuration="1.877287994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:18.876051298 +0000 UTC m=+1475.040452942" watchObservedRunningTime="2026-10-30 20:06:18.877287994 +0000 UTC m=+1475.041689640"
Oct 30 20:06:19.516796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.515834    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:19.534854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.534505    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:19.534854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.534565    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:19.592975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.592924    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.670802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.669215    4917 generic.go:356] "Generic (PLEG): container finished" podID="1f9c91d7-abb9-4731-a42c-2c7961a1de9a" containerID="3508104b270cdab46c31bc097ad6928d19ff8e5b73044930de0a28f12f95b53e" exitCode=0
Oct 30 20:06:19.671004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.670842    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9" event={"ID":"1f9c91d7-abb9-4731-a42c-2c7961a1de9a","Type":"ContainerDied","Data":"3508104b270cdab46c31bc097ad6928d19ff8e5b73044930de0a28f12f95b53e"}
Oct 30 20:06:19.701799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.700862    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:06:19.731888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.731847    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.772966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.772874    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.840801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.838864    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.894797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.883922    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.904794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.900344    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.904794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.901585    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 20:06:19.924796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:19.921663    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.036843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:20.036131    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.514007 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:20.513972    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:20.514181 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:20.514041    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:20.907926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:20.907895    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:21.009593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.009554    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl\" (UniqueName: \"kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl\") pod \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") "
Oct 30 20:06:21.009810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.009717    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca\") pod \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") "
Oct 30 20:06:21.009810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.009803    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca\") pod \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\" (UID: \"1f9c91d7-abb9-4731-a42c-2c7961a1de9a\") "
Oct 30 20:06:21.010467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.010436    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca" pod "1f9c91d7-abb9-4731-a42c-2c7961a1de9a" (UID: "1f9c91d7-abb9-4731-a42c-2c7961a1de9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:21.012823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.012793    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca" pod "1f9c91d7-abb9-4731-a42c-2c7961a1de9a" (UID: "1f9c91d7-abb9-4731-a42c-2c7961a1de9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:21.012930 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.012904    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl" pod "1f9c91d7-abb9-4731-a42c-2c7961a1de9a" (UID: "1f9c91d7-abb9-4731-a42c-2c7961a1de9a"). InnerVolumeSpecName "kube-api-access-8cbnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:21.111970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.111939    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbnl\" (UniqueName: \"kubernetes.io/projected/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-kube-api-access-8cbnl\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:21.111970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.111967    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:21.111970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.111977    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9c91d7-abb9-4731-a42c-2c7961a1de9a-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:21.513844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.513810    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:21.514032 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.513866    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:21.689717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.689682    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"
Oct 30 20:06:21.689717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.689707    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9" event={"ID":"1f9c91d7-abb9-4731-a42c-2c7961a1de9a","Type":"ContainerDied","Data":"9b8b6ac5574b6863126cd012f8d1ea4b1a465b2f434bec9af4d2a726f37cb8e9"}
Oct 30 20:06:21.690003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:21.689753    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8b6ac5574b6863126cd012f8d1ea4b1a465b2f434bec9af4d2a726f37cb8e9"
Oct 30 20:06:22.513930 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:22.513890    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:22.514327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:22.513959    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:23.514444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:23.514409    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:23.514891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:23.514461    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:24.513999 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:24.513960    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:24.514187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:24.514026    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:25.514194 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:25.514161    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:06:25.514575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:25.514223    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:06:26.518099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:26.518067    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:26.518504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:26.518486    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:26.519155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:26.519135    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:06:32.045223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.045187    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"]
Oct 30 20:06:32.045783 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.045751    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1f9c91d7-abb9-4731-a42c-2c7961a1de9a" containerName="revoke-keys"
Oct 30 20:06:32.045885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.045785    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9c91d7-abb9-4731-a42c-2c7961a1de9a" containerName="revoke-keys"
Oct 30 20:06:32.045931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.045909    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9c91d7-abb9-4731-a42c-2c7961a1de9a" containerName="revoke-keys"
Oct 30 20:06:32.049581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.049563    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.058114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.058092    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"]
Oct 30 20:06:32.095054 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.095029    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fa7c74-cm8d6"]
Oct 30 20:06:32.104243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.104221    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:32.104419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.104319    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:32.104419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.104412    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:32.108743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.108712    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"]
Oct 30 20:06:32.141621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.141593    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:32.172805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.172742    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.172974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.172897    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtzc\" (UniqueName: \"kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.172974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.172928    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.274361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.274330    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.274553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.274431    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtzc\" (UniqueName: \"kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.274553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.274461    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.275308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.275281    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.277155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.277131    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.336741 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.336680    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtzc\" (UniqueName: \"kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc\") pod \"maas-api-revoke-sub-2d76a077-x6pgv\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.363532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.363500    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:32.598196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.598119    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"]
Oct 30 20:06:32.602321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.602303    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679373    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679429    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679556    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679653    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679687    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5xd\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679843    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.679916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.679901    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.680042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.680025    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.680099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.680084    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.709278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.709246    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"]
Oct 30 20:06:32.765785 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.765746    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"]
Oct 30 20:06:32.776437 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.776412    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv" event={"ID":"b48b781b-956d-4513-9234-820c801a62c4","Type":"ContainerStarted","Data":"f3841f38a5043fa95a056909c871958374679306d01731d7c2a880d70330103f"}
Oct 30 20:06:32.781020 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.780995    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781048    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781139    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781190    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781260    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781294    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781347    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781378    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781416    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781442    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5xd\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781646    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781830    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781912 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781879    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.781912 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.781895    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.783730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.783709    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.783851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.783835    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.843734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.843700    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.843897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.843824    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5xd\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:32.916548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:32.916467    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:33.357744 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:06:33.357714    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f3a3f_fd98_433c_b9d9_0bda69e77a7b.slice/crio-8a36d53670ed9d6c4ee16ecb604e3b0bf3d07da793f2aaf842550bafc6980e7c WatchSource:0}: Error finding container 8a36d53670ed9d6c4ee16ecb604e3b0bf3d07da793f2aaf842550bafc6980e7c: Status 404 returned error can't find the container with id 8a36d53670ed9d6c4ee16ecb604e3b0bf3d07da793f2aaf842550bafc6980e7c
Oct 30 20:06:33.358101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.357814    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"]
Oct 30 20:06:33.360499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.360463    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:33.360578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.360548    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:33.360613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.360589    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:33.785618 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.785559    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" event={"ID":"405f3a3f-fd98-433c-b9d9-0bda69e77a7b","Type":"ContainerStarted","Data":"7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2"}
Oct 30 20:06:33.785618 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.785622    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" event={"ID":"405f3a3f-fd98-433c-b9d9-0bda69e77a7b","Type":"ContainerStarted","Data":"8a36d53670ed9d6c4ee16ecb604e3b0bf3d07da793f2aaf842550bafc6980e7c"}
Oct 30 20:06:33.787368 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.787340    4917 generic.go:356] "Generic (PLEG): container finished" podID="b48b781b-956d-4513-9234-820c801a62c4" containerID="209bc1ff114be921e9fff8e2383fc730105ba361a25bf47f22aedc5a1b09b4b9" exitCode=0
Oct 30 20:06:33.787482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.787403    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv" event={"ID":"b48b781b-956d-4513-9234-820c801a62c4","Type":"ContainerDied","Data":"209bc1ff114be921e9fff8e2383fc730105ba361a25bf47f22aedc5a1b09b4b9"}
Oct 30 20:06:33.913317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.913200    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podStartSLOduration=1.913185364 podStartE2EDuration="1.913185364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:33.907527539 +0000 UTC m=+1490.071929192" watchObservedRunningTime="2026-10-30 20:06:33.913185364 +0000 UTC m=+1490.077587007"
Oct 30 20:06:33.917340 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:33.917301    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:34.003419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:34.003387    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff6f255-9917-4dc9-ba01-15b6e115e063" path="/var/lib/kubelet/pods/5ff6f255-9917-4dc9-ba01-15b6e115e063/volumes"
Oct 30 20:06:34.917859 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:34.917829    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:06:34.918346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:34.917895    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:06:34.993278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:34.993256    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:35.111463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.111368    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca\") pod \"b48b781b-956d-4513-9234-820c801a62c4\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") "
Oct 30 20:06:35.111463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.111425    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc\" (UniqueName: \"kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc\") pod \"b48b781b-956d-4513-9234-820c801a62c4\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") "
Oct 30 20:06:35.111694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.111479    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca\") pod \"b48b781b-956d-4513-9234-820c801a62c4\" (UID: \"b48b781b-956d-4513-9234-820c801a62c4\") "
Oct 30 20:06:35.111819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.111795    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca" pod "b48b781b-956d-4513-9234-820c801a62c4" (UID: "b48b781b-956d-4513-9234-820c801a62c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:35.112158 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.112126    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48b781b-956d-4513-9234-820c801a62c4-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:35.113718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.113694    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca" pod "b48b781b-956d-4513-9234-820c801a62c4" (UID: "b48b781b-956d-4513-9234-820c801a62c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:35.113839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.113745    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc" pod "b48b781b-956d-4513-9234-820c801a62c4" (UID: "b48b781b-956d-4513-9234-820c801a62c4"). InnerVolumeSpecName "kube-api-access-5vtzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:35.213605 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.213569    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtzc\" (UniqueName: \"kubernetes.io/projected/b48b781b-956d-4513-9234-820c801a62c4-kube-api-access-5vtzc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:35.213605 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.213601    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48b781b-956d-4513-9234-820c801a62c4-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:35.802560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.802528    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"
Oct 30 20:06:35.802560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.802549    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv" event={"ID":"b48b781b-956d-4513-9234-820c801a62c4","Type":"ContainerDied","Data":"f3841f38a5043fa95a056909c871958374679306d01731d7c2a880d70330103f"}
Oct 30 20:06:35.802802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.802598    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3841f38a5043fa95a056909c871958374679306d01731d7c2a880d70330103f"
Oct 30 20:06:35.917221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.917187    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:06:35.917401 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:35.917251    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:06:36.917278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:36.917241    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:06:36.917672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:36.917323    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:06:37.916947 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:37.916913    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:06:37.917126 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:37.916966    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:06:38.069575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.069537    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"]
Oct 30 20:06:38.070221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.070202    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b48b781b-956d-4513-9234-820c801a62c4" containerName="revoke-keys"
Oct 30 20:06:38.070266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.070223    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48b781b-956d-4513-9234-820c801a62c4" containerName="revoke-keys"
Oct 30 20:06:38.070328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.070319    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48b781b-956d-4513-9234-820c801a62c4" containerName="revoke-keys"
Oct 30 20:06:38.074008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.073982    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.123953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.123921    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"]
Oct 30 20:06:38.167521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.167419    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.167521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.167482    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xm2\" (UniqueName: \"kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.167743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.167655    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.169667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.169634    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:38.169667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.169643    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:38.169914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.169894    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:38.169914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.169902    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:38.269470 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.269440    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.269655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.269484    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xm2\" (UniqueName: \"kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.269655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.269536    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.270323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.270296    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.271935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.271914    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.332274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.332241    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xm2\" (UniqueName: \"kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2\") pod \"maas-api-revoke-sub-b2ea0add-9rk45\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.388294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.388255    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:38.785943 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:06:38.785907    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825f4298_0eb6_4193_8a15_5d5a11703b0a.slice/crio-1dccbaa1dc50e6be8cd527fd786749d604cc7efdd4f522a0c839f37a93ae5ba7 WatchSource:0}: Error finding container 1dccbaa1dc50e6be8cd527fd786749d604cc7efdd4f522a0c839f37a93ae5ba7: Status 404 returned error can't find the container with id 1dccbaa1dc50e6be8cd527fd786749d604cc7efdd4f522a0c839f37a93ae5ba7
Oct 30 20:06:38.787227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.787187    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"]
Oct 30 20:06:38.825616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.825588    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45" event={"ID":"825f4298-0eb6-4193-8a15-5d5a11703b0a","Type":"ContainerStarted","Data":"1dccbaa1dc50e6be8cd527fd786749d604cc7efdd4f522a0c839f37a93ae5ba7"}
Oct 30 20:06:38.917717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.917678    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:06:38.917905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:38.917741    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:06:39.681423 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:06:39.681380    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:06:39.837317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:39.837283    4917 generic.go:356] "Generic (PLEG): container finished" podID="825f4298-0eb6-4193-8a15-5d5a11703b0a" containerID="7c37f0d5aaede54b73f3fc1d7b1963ac9ad47d59f98e9fbc31ca678a17fb14fd" exitCode=0
Oct 30 20:06:39.837510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:39.837493    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45" event={"ID":"825f4298-0eb6-4193-8a15-5d5a11703b0a","Type":"ContainerDied","Data":"7c37f0d5aaede54b73f3fc1d7b1963ac9ad47d59f98e9fbc31ca678a17fb14fd"}
Oct 30 20:06:39.917495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:39.917463    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:06:39.917669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:39.917514    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:06:40.706545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.706514    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"]
Oct 30 20:06:40.710821 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.710799    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.805943 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.805913    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.806124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.805956    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.806124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.805971    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.806124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.806043    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.806241 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.806158    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgwd2\" (UniqueName: \"kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.806241 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.806220    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.811812 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.811780    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"distinct-2-w1-3ccc9f-kserve-self-signed-certs\""
Oct 30 20:06:40.816162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.816141    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"]
Oct 30 20:06:40.908344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.908306    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.908566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.908370    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.908566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.908399    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.908566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.908422    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.908566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.908515    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwd2\" (UniqueName: \"kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.908855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.908602    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.909207 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.909179    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.909450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.909429    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.910042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.910012    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.915547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.915523    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.921200 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.921172    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:40.952409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.952187    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:40.957208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.953629    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:40.962226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.959082    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:06:40.972944 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:40.972921    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwd2\" (UniqueName: \"kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2\") pod \"distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:41.023824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.023690    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:41.047582 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.047549    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:41.113983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.113142    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca\") pod \"825f4298-0eb6-4193-8a15-5d5a11703b0a\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") "
Oct 30 20:06:41.113983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.113283    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2\" (UniqueName: \"kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2\") pod \"825f4298-0eb6-4193-8a15-5d5a11703b0a\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") "
Oct 30 20:06:41.113983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.113436    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca\") pod \"825f4298-0eb6-4193-8a15-5d5a11703b0a\" (UID: \"825f4298-0eb6-4193-8a15-5d5a11703b0a\") "
Oct 30 20:06:41.113983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.113921    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca" pod "825f4298-0eb6-4193-8a15-5d5a11703b0a" (UID: "825f4298-0eb6-4193-8a15-5d5a11703b0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:41.117204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.117173    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca" pod "825f4298-0eb6-4193-8a15-5d5a11703b0a" (UID: "825f4298-0eb6-4193-8a15-5d5a11703b0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:41.117853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.117828    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2" pod "825f4298-0eb6-4193-8a15-5d5a11703b0a" (UID: "825f4298-0eb6-4193-8a15-5d5a11703b0a"). InnerVolumeSpecName "kube-api-access-82xm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:41.215543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.215449    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:41.215764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.215744    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xm2\" (UniqueName: \"kubernetes.io/projected/825f4298-0eb6-4193-8a15-5d5a11703b0a-kube-api-access-82xm2\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:41.215965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.215942    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825f4298-0eb6-4193-8a15-5d5a11703b0a-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:41.423243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.423203    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"]
Oct 30 20:06:41.423991 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:06:41.423963    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb1b3140_5033_4b55_be7e_ea71cf750811.slice/crio-f8a50e53f1d1454cf4d514d4b74e7520bf7fcd02662733344f85114c2e85005b WatchSource:0}: Error finding container f8a50e53f1d1454cf4d514d4b74e7520bf7fcd02662733344f85114c2e85005b: Status 404 returned error can't find the container with id f8a50e53f1d1454cf4d514d4b74e7520bf7fcd02662733344f85114c2e85005b
Oct 30 20:06:41.858160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.858059    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" event={"ID":"cb1b3140-5033-4b55-be7e-ea71cf750811","Type":"ContainerStarted","Data":"89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85"}
Oct 30 20:06:41.858160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.858111    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" event={"ID":"cb1b3140-5033-4b55-be7e-ea71cf750811","Type":"ContainerStarted","Data":"f8a50e53f1d1454cf4d514d4b74e7520bf7fcd02662733344f85114c2e85005b"}
Oct 30 20:06:41.859555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.859538    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"
Oct 30 20:06:41.859652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.859581    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45" event={"ID":"825f4298-0eb6-4193-8a15-5d5a11703b0a","Type":"ContainerDied","Data":"1dccbaa1dc50e6be8cd527fd786749d604cc7efdd4f522a0c839f37a93ae5ba7"}
Oct 30 20:06:41.859652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.859625    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dccbaa1dc50e6be8cd527fd786749d604cc7efdd4f522a0c839f37a93ae5ba7"
Oct 30 20:06:41.978665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:41.978613    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podStartSLOduration=1.978597647 podStartE2EDuration="1.978597647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:41.976252633 +0000 UTC m=+1498.140654274" watchObservedRunningTime="2026-10-30 20:06:41.978597647 +0000 UTC m=+1498.142999283"
Oct 30 20:06:42.024695 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:42.024661    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:44.174476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:44.174448    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:06:44.195892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:44.195857    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:06:46.425839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.425803    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"]
Oct 30 20:06:46.426367 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.426354    4917 cpu_manager.go:414] "containerMap: removing container" podUID="825f4298-0eb6-4193-8a15-5d5a11703b0a" containerName="revoke-keys"
Oct 30 20:06:46.426426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.426368    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825f4298-0eb6-4193-8a15-5d5a11703b0a" containerName="revoke-keys"
Oct 30 20:06:46.426481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.426473    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="825f4298-0eb6-4193-8a15-5d5a11703b0a" containerName="revoke-keys"
Oct 30 20:06:46.431129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.431111    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.560850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.560816    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"]
Oct 30 20:06:46.586147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586079    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46kg\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586138    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586391    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586491 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586463    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586491 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586486    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586540    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586603    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586621    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.586759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.586658    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.687814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.687747    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.687959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.687823    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.687959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.687843    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.687959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.687889    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688005    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688035    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688085    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688103    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688196    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688243    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688254    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46kg\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688294    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688355    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.688945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.688928    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.690334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.690307    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.690575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.690555    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.750023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.749988    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:46.750184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:46.750000    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46kg\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-smzf7\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:47.045101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.045021    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:47.057089 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.057060    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"]
Oct 30 20:06:47.068991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.068962    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899e4914-7nth9"]
Oct 30 20:06:47.495346 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:06:47.495314    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6a27ec_5567_4684_b862_478385943f99.slice/crio-33b5b58580ca9bba57709dc1e1b52735e2dbff79a2f9bdac5ec0b7401e6e2a0b WatchSource:0}: Error finding container 33b5b58580ca9bba57709dc1e1b52735e2dbff79a2f9bdac5ec0b7401e6e2a0b: Status 404 returned error can't find the container with id 33b5b58580ca9bba57709dc1e1b52735e2dbff79a2f9bdac5ec0b7401e6e2a0b
Oct 30 20:06:47.495988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.495964    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"]
Oct 30 20:06:47.498080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.498044    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:47.498155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.498112    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:47.498155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.498140    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:06:47.906563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.906471    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" event={"ID":"6c6a27ec-5567-4684-b862-478385943f99","Type":"ContainerStarted","Data":"0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28"}
Oct 30 20:06:47.906563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.906539    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" event={"ID":"6c6a27ec-5567-4684-b862-478385943f99","Type":"ContainerStarted","Data":"33b5b58580ca9bba57709dc1e1b52735e2dbff79a2f9bdac5ec0b7401e6e2a0b"}
Oct 30 20:06:47.945281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:47.945248    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:06:48.005271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:48.005240    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045d4486-1800-4482-bcdd-7a1dd2570fbe" path="/var/lib/kubelet/pods/045d4486-1800-4482-bcdd-7a1dd2570fbe/volumes"
Oct 30 20:06:48.038662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:48.038601    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podStartSLOduration=2.038586081 podStartE2EDuration="2.038586081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:48.035114438 +0000 UTC m=+1504.199516082" watchObservedRunningTime="2026-10-30 20:06:48.038586081 +0000 UTC m=+1504.202987723"
Oct 30 20:06:48.051797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:48.048540    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:48.065794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:48.057914    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:48.065794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:48.057984    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:49.059797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:49.050874    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:49.059797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:49.050939    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:50.051276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.051234    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:50.051590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.051558    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:50.610795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.609925    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.719962 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.719903    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.745061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.745018    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:06:50.746331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.746286    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.893022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.892906    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.894239 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.894193    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.926084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:50.926044    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.028204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.028171    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:51.041669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.041623    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.047252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.047215    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:51.047548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.047520    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:51.086250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.086153    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:06:51.222501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.222469    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:51.259916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.259869    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded"
Oct 30 20:06:51.318795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.317860    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.612099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.611999    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.617804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.617684    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:06:51.724174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:51.724141    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:06:52.045809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:52.045041    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:06:52.049214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:52.049182    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:52.049447 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:52.049419    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:53.065799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:53.064723    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:53.065799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:53.064803    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:53.072810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:53.072761    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"]
Oct 30 20:06:53.121728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:53.121694    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26bc6b8-vmdbt"]
Oct 30 20:06:54.006504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:54.006467    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a4681c-bd23-4556-aae3-f2a6c6ef3653" path="/var/lib/kubelet/pods/72a4681c-bd23-4556-aae3-f2a6c6ef3653/volumes"
Oct 30 20:06:54.046630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:54.046594    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:06:54.046820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:54.046646    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:06:55.286601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.286553    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"]
Oct 30 20:06:55.293672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.293642    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.349890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.349853    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"]
Oct 30 20:06:55.358822 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.358757    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxj9\" (UniqueName: \"kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.359013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.358987    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.359096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.359030    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.395080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.395042    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:55.395257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.395101    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:55.395257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.395113    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:55.397505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.397479    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:55.460208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.460179    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxj9\" (UniqueName: \"kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.460396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.460287    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.460396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.460319    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.460970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.460950    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.462979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.462953    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.522408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.522369    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxj9\" (UniqueName: \"kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9\") pod \"maas-api-revoke-sub-d558de14-d555l\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:55.611082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:55.611003    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:56.009883 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:06:56.009839    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81cb4e0_c353_4938_82c2_b5953f1a49f2.slice/crio-a3a631b39e0ea63fed76561fbcaf6219e6865415cd997fab1fa81a2ffc0a39cc WatchSource:0}: Error finding container a3a631b39e0ea63fed76561fbcaf6219e6865415cd997fab1fa81a2ffc0a39cc: Status 404 returned error can't find the container with id a3a631b39e0ea63fed76561fbcaf6219e6865415cd997fab1fa81a2ffc0a39cc
Oct 30 20:06:56.010868 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.010837    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"]
Oct 30 20:06:56.085949 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.085911    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:56.086138 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.085973    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:56.111196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.111164    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l" event={"ID":"c81cb4e0-c353-4938-82c2-b5953f1a49f2","Type":"ContainerStarted","Data":"a3a631b39e0ea63fed76561fbcaf6219e6865415cd997fab1fa81a2ffc0a39cc"}
Oct 30 20:06:56.567832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.567802    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:56.568327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.568296    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:56.569624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:56.569602    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:06:57.126528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:57.126488    4917 generic.go:356] "Generic (PLEG): container finished" podID="c81cb4e0-c353-4938-82c2-b5953f1a49f2" containerID="b36532585f2b425b6efe8cf77d1eaea59edcaf92ff7ee40594df0d4c5c8b88d8" exitCode=0
Oct 30 20:06:57.126700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:57.126572    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l" event={"ID":"c81cb4e0-c353-4938-82c2-b5953f1a49f2","Type":"ContainerDied","Data":"b36532585f2b425b6efe8cf77d1eaea59edcaf92ff7ee40594df0d4c5c8b88d8"}
Oct 30 20:06:58.313820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.313790    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:58.400671 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.400589    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9\" (UniqueName: \"kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9\") pod \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") "
Oct 30 20:06:58.400870 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.400731    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca\") pod \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") "
Oct 30 20:06:58.400952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.400880    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca\") pod \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\" (UID: \"c81cb4e0-c353-4938-82c2-b5953f1a49f2\") "
Oct 30 20:06:58.401153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.401124    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca" pod "c81cb4e0-c353-4938-82c2-b5953f1a49f2" (UID: "c81cb4e0-c353-4938-82c2-b5953f1a49f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:58.401380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.401360    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:58.402763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.402744    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9" pod "c81cb4e0-c353-4938-82c2-b5953f1a49f2" (UID: "c81cb4e0-c353-4938-82c2-b5953f1a49f2"). InnerVolumeSpecName "kube-api-access-wgxj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:58.402988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.402865    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca" pod "c81cb4e0-c353-4938-82c2-b5953f1a49f2" (UID: "c81cb4e0-c353-4938-82c2-b5953f1a49f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:58.502402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.502367    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81cb4e0-c353-4938-82c2-b5953f1a49f2-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:58.502402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:58.502398    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgxj9\" (UniqueName: \"kubernetes.io/projected/c81cb4e0-c353-4938-82c2-b5953f1a49f2-kube-api-access-wgxj9\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:59.065493 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:59.065457    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"]
Oct 30 20:06:59.115090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:59.115059    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c314d64-dc7vz"]
Oct 30 20:06:59.143381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:59.143348    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"
Oct 30 20:06:59.143381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:59.143377    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l" event={"ID":"c81cb4e0-c353-4938-82c2-b5953f1a49f2","Type":"ContainerDied","Data":"a3a631b39e0ea63fed76561fbcaf6219e6865415cd997fab1fa81a2ffc0a39cc"}
Oct 30 20:06:59.143567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:06:59.143419    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a631b39e0ea63fed76561fbcaf6219e6865415cd997fab1fa81a2ffc0a39cc"
Oct 30 20:07:00.008574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:00.008535    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6efee9-7d79-4460-92a1-79c622e7c3c1" path="/var/lib/kubelet/pods/ff6efee9-7d79-4460-92a1-79c622e7c3c1/volumes"
Oct 30 20:07:01.535349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.535312    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"]
Oct 30 20:07:01.535887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.535872    4917 cpu_manager.go:414] "containerMap: removing container" podUID="c81cb4e0-c353-4938-82c2-b5953f1a49f2" containerName="revoke-keys"
Oct 30 20:07:01.535940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.535888    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81cb4e0-c353-4938-82c2-b5953f1a49f2" containerName="revoke-keys"
Oct 30 20:07:01.536013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.536004    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81cb4e0-c353-4938-82c2-b5953f1a49f2" containerName="revoke-keys"
Oct 30 20:07:01.539807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.539763    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640065    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxksj\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640113    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640245    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640325    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640387    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640493    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640557    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640610    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.640683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.640649    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.650722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.650684    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"]
Oct 30 20:07:01.742372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742335    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742469    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742514    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxksj\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742536    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742592    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742634    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742689    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742826    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.742972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.742892    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.743100 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.743065    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.743237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.743126    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.743237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.743180    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.743441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.743418    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.743557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.743540    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.745110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.745090    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.745674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.745655    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.805130 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.805053    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.805368 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.805342    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxksj\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-bgwln\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.853889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.853855    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:01.924030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.923811    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"]
Oct 30 20:07:01.929853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.929827    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:01.984904 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.984871    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"]
Oct 30 20:07:01.985265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.985244    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:01.985445 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.985430    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:01.985524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.985485    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:01.985948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:01.985909    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:02.048458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.048423    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.048636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.048464    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.048749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.048726    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zth\" (UniqueName: \"kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.150488 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.150398    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zth\" (UniqueName: \"kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.150666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.150573    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.150666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.150605    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.151246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.151225    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.153035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.153018    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.213148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.213115    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zth\" (UniqueName: \"kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth\") pod \"maas-api-revoke-sub-f2771248-7xwnc\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.252005 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.251975    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:02.301544 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:02.301514    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae71a5c_9975_46d6_8a0d_5392b006b42f.slice/crio-ab3636f0b193eb7c4cd34b1261e501a9e43a658350728e5a514b764bcf7443ca WatchSource:0}: Error finding container ab3636f0b193eb7c4cd34b1261e501a9e43a658350728e5a514b764bcf7443ca: Status 404 returned error can't find the container with id ab3636f0b193eb7c4cd34b1261e501a9e43a658350728e5a514b764bcf7443ca
Oct 30 20:07:02.302625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.302599    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"]
Oct 30 20:07:02.304554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.304518    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:02.304664 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.304603    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:02.304664 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.304650    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:02.634210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:02.634176    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"]
Oct 30 20:07:02.634644 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:02.634186    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed9ecde_4773_47d7_b291_2376a50d1063.slice/crio-7040de44e247a2bc6cf4d52f24be4c75684fbb8ad3bc2841b3c0823243f75306 WatchSource:0}: Error finding container 7040de44e247a2bc6cf4d52f24be4c75684fbb8ad3bc2841b3c0823243f75306: Status 404 returned error can't find the container with id 7040de44e247a2bc6cf4d52f24be4c75684fbb8ad3bc2841b3c0823243f75306
Oct 30 20:07:03.122082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.122044    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"]
Oct 30 20:07:03.126253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.126226    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.197461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.197424    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" event={"ID":"4ae71a5c-9975-46d6-8a0d-5392b006b42f","Type":"ContainerStarted","Data":"c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b"}
Oct 30 20:07:03.197706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.197683    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" event={"ID":"4ae71a5c-9975-46d6-8a0d-5392b006b42f","Type":"ContainerStarted","Data":"ab3636f0b193eb7c4cd34b1261e501a9e43a658350728e5a514b764bcf7443ca"}
Oct 30 20:07:03.199461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.199437    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc" event={"ID":"1ed9ecde-4773-47d7-b291-2376a50d1063","Type":"ContainerStarted","Data":"b664dbab43abf58fe0186abf36bc973fb2b6753c0d9e6ae455fbe666e5b212df"}
Oct 30 20:07:03.199574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.199478    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc" event={"ID":"1ed9ecde-4773-47d7-b291-2376a50d1063","Type":"ContainerStarted","Data":"7040de44e247a2bc6cf4d52f24be4c75684fbb8ad3bc2841b3c0823243f75306"}
Oct 30 20:07:03.217038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.217004    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"]
Oct 30 20:07:03.273872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.273735    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.273872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.273863    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.273929    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.273970    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.273993    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wj9z\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274283 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.274243    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.274585    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.274640    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.274749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.274695    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376088    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376178    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376220    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376242    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wj9z\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376287    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376355    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376390    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376425    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376563    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376701    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376865    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.376971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.376946    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.377033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.377003    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.377195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.377167    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.379041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.379017    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.379372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.379352    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.394914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.394863    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc" podStartSLOduration=2.394849938 podStartE2EDuration="2.394849938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:03.390652849 +0000 UTC m=+1519.555054513" watchObservedRunningTime="2026-10-30 20:07:03.394849938 +0000 UTC m=+1519.559251580"
Oct 30 20:07:03.438540 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.438509    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.438682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.438551    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wj9z\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.460049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.460012    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:03.532464 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.532424    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podStartSLOduration=2.532399688 podStartE2EDuration="2.532399688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:03.530134476 +0000 UTC m=+1519.694536118" watchObservedRunningTime="2026-10-30 20:07:03.532399688 +0000 UTC m=+1519.696801354"
Oct 30 20:07:03.863015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.862932    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:03.863946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.863604    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:07:03.863946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.863656    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 20:07:03.902310 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:03.902279    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd641ab76_a95f_4043_8e38_d6a7b8323900.slice/crio-ea5f0c44031f6aba9d176763ddb87e83154221c8d955038de1a5bc84c207c6cc WatchSource:0}: Error finding container ea5f0c44031f6aba9d176763ddb87e83154221c8d955038de1a5bc84c207c6cc: Status 404 returned error can't find the container with id ea5f0c44031f6aba9d176763ddb87e83154221c8d955038de1a5bc84c207c6cc
Oct 30 20:07:03.903578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.903549    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"]
Oct 30 20:07:03.905282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.905250    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:03.905377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.905315    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:03.905377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:03.905344    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:04.075990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.075896    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"]
Oct 30 20:07:04.080365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.080336    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-458882e7-rpspn"]
Oct 30 20:07:04.209136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.209079    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" event={"ID":"d641ab76-a95f-4043-8e38-d6a7b8323900","Type":"ContainerStarted","Data":"7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212"}
Oct 30 20:07:04.209330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.209152    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" event={"ID":"d641ab76-a95f-4043-8e38-d6a7b8323900","Type":"ContainerStarted","Data":"ea5f0c44031f6aba9d176763ddb87e83154221c8d955038de1a5bc84c207c6cc"}
Oct 30 20:07:04.211184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.211157    4917 generic.go:356] "Generic (PLEG): container finished" podID="1ed9ecde-4773-47d7-b291-2376a50d1063" containerID="b664dbab43abf58fe0186abf36bc973fb2b6753c0d9e6ae455fbe666e5b212df" exitCode=0
Oct 30 20:07:04.211306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.211216    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc" event={"ID":"1ed9ecde-4773-47d7-b291-2376a50d1063","Type":"ContainerDied","Data":"b664dbab43abf58fe0186abf36bc973fb2b6753c0d9e6ae455fbe666e5b212df"}
Oct 30 20:07:04.336645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.336535    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podStartSLOduration=1.336520768 podStartE2EDuration="1.336520768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:04.335043022 +0000 UTC m=+1520.499444664" watchObservedRunningTime="2026-10-30 20:07:04.336520768 +0000 UTC m=+1520.500922410"
Oct 30 20:07:04.468594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.467445    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:04.865793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.864878    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:07:04.865793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:04.864940    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 20:07:05.499360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.498919    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 20:07:05.499360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.498970    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 20:07:05.509234 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.509206    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:05.610453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.610416    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca\") pod \"1ed9ecde-4773-47d7-b291-2376a50d1063\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") "
Oct 30 20:07:05.610636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.610540    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca\") pod \"1ed9ecde-4773-47d7-b291-2376a50d1063\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") "
Oct 30 20:07:05.610636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.610607    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth\" (UniqueName: \"kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth\") pod \"1ed9ecde-4773-47d7-b291-2376a50d1063\" (UID: \"1ed9ecde-4773-47d7-b291-2376a50d1063\") "
Oct 30 20:07:05.611190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.611165    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca" pod "1ed9ecde-4773-47d7-b291-2376a50d1063" (UID: "1ed9ecde-4773-47d7-b291-2376a50d1063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:05.611419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.611405    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:05.612967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.612938    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca" pod "1ed9ecde-4773-47d7-b291-2376a50d1063" (UID: "1ed9ecde-4773-47d7-b291-2376a50d1063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:05.613064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.612967    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth" pod "1ed9ecde-4773-47d7-b291-2376a50d1063" (UID: "1ed9ecde-4773-47d7-b291-2376a50d1063"). InnerVolumeSpecName "kube-api-access-82zth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:05.713207 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.713170    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed9ecde-4773-47d7-b291-2376a50d1063-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:05.713378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.713208    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zth\" (UniqueName: \"kubernetes.io/projected/1ed9ecde-4773-47d7-b291-2376a50d1063-kube-api-access-82zth\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:05.866849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.866207    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:07:05.866849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:05.866273    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 20:07:06.004515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.004479    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079348fe-cdd8-424e-aa73-7a655fbd13fa" path="/var/lib/kubelet/pods/079348fe-cdd8-424e-aa73-7a655fbd13fa/volumes"
Oct 30 20:07:06.228083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.227971    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"
Oct 30 20:07:06.228083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.228017    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc" event={"ID":"1ed9ecde-4773-47d7-b291-2376a50d1063","Type":"ContainerDied","Data":"7040de44e247a2bc6cf4d52f24be4c75684fbb8ad3bc2841b3c0823243f75306"}
Oct 30 20:07:06.228083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.228060    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7040de44e247a2bc6cf4d52f24be4c75684fbb8ad3bc2841b3c0823243f75306"
Oct 30 20:07:06.463623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.463183    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 20:07:06.463623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.463247    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 20:07:06.860794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.857189    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:07:06.860794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:06.857254    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 20:07:07.464795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:07.463197    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 20:07:07.464795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:07.463262    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 20:07:07.864849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:07.863876    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:07:07.864849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:07.863954    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 20:07:08.463795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:08.463196    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 20:07:08.463795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:08.463283    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 20:07:08.860144 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:08.860049    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:07:08.860691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:08.860660    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 20:07:09.463794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:09.463191    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 20:07:09.463794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:09.463256    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 20:07:09.697698 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:07:09.697633    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:07:09.976537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:09.976082    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:09.977142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:09.976622    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:09.977481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:09.977439    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:07:10.460372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.460337    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 20:07:10.460546 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.460400    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 20:07:10.944548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.944515    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"]
Oct 30 20:07:10.945186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.945166    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1ed9ecde-4773-47d7-b291-2376a50d1063" containerName="revoke-keys"
Oct 30 20:07:10.945312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.945187    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed9ecde-4773-47d7-b291-2376a50d1063" containerName="revoke-keys"
Oct 30 20:07:10.945392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.945376    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed9ecde-4773-47d7-b291-2376a50d1063" containerName="revoke-keys"
Oct 30 20:07:10.950024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:10.950002    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.004693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.004662    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:11.005080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.004789    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:11.005080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.004873    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:11.007930 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.007903    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"]
Oct 30 20:07:11.051740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.051709    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:11.111458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.111425    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.111659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.111635    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.111902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.111880    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb99v\" (UniqueName: \"kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.213423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.213338    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.213591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.213520    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb99v\" (UniqueName: \"kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.213591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.213575    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.214321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.214291    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.216140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.216117    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.276326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.276292    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb99v\" (UniqueName: \"kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v\") pod \"maas-api-revoke-sub-ce4a87a0-hq67l\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.536114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.536033    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:11.536501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.536473    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:11.537509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.537485    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:07:11.564671 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.564637    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:11.966039 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:11.965998    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode312f4f5_febd_4877_b571_8e4c8000699d.slice/crio-5cc55ea3695c7de0cdc24650aba236df3838c35419493aba423f99e8741da733 WatchSource:0}: Error finding container 5cc55ea3695c7de0cdc24650aba236df3838c35419493aba423f99e8741da733: Status 404 returned error can't find the container with id 5cc55ea3695c7de0cdc24650aba236df3838c35419493aba423f99e8741da733
Oct 30 20:07:11.966174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:11.966056    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"]
Oct 30 20:07:12.300356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:12.300322    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l" event={"ID":"e312f4f5-febd-4877-b571-8e4c8000699d","Type":"ContainerStarted","Data":"5cc55ea3695c7de0cdc24650aba236df3838c35419493aba423f99e8741da733"}
Oct 30 20:07:13.309509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:13.309470    4917 generic.go:356] "Generic (PLEG): container finished" podID="e312f4f5-febd-4877-b571-8e4c8000699d" containerID="b85f5d3a8f3aa5f662611adabef31db650c7d00df8ea6eff571bddaf29e55853" exitCode=0
Oct 30 20:07:13.309917 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:13.309554    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l" event={"ID":"e312f4f5-febd-4877-b571-8e4c8000699d","Type":"ContainerDied","Data":"b85f5d3a8f3aa5f662611adabef31db650c7d00df8ea6eff571bddaf29e55853"}
Oct 30 20:07:14.469255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.469222    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:14.562843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.562812    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca\") pod \"e312f4f5-febd-4877-b571-8e4c8000699d\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") "
Oct 30 20:07:14.562997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.562985    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca\") pod \"e312f4f5-febd-4877-b571-8e4c8000699d\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") "
Oct 30 20:07:14.563057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.563022    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v\" (UniqueName: \"kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v\") pod \"e312f4f5-febd-4877-b571-8e4c8000699d\" (UID: \"e312f4f5-febd-4877-b571-8e4c8000699d\") "
Oct 30 20:07:14.563209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.563179    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca" pod "e312f4f5-febd-4877-b571-8e4c8000699d" (UID: "e312f4f5-febd-4877-b571-8e4c8000699d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:14.563537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.563515    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:14.565145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.565116    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca" pod "e312f4f5-febd-4877-b571-8e4c8000699d" (UID: "e312f4f5-febd-4877-b571-8e4c8000699d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:14.565226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.565184    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v" pod "e312f4f5-febd-4877-b571-8e4c8000699d" (UID: "e312f4f5-febd-4877-b571-8e4c8000699d"). InnerVolumeSpecName "kube-api-access-vb99v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:14.664350 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.664319    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e312f4f5-febd-4877-b571-8e4c8000699d-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:14.664350 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:14.664347    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb99v\" (UniqueName: \"kubernetes.io/projected/e312f4f5-febd-4877-b571-8e4c8000699d-kube-api-access-vb99v\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:15.328305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:15.328259    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l" event={"ID":"e312f4f5-febd-4877-b571-8e4c8000699d","Type":"ContainerDied","Data":"5cc55ea3695c7de0cdc24650aba236df3838c35419493aba423f99e8741da733"}
Oct 30 20:07:15.328305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:15.328299    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"
Oct 30 20:07:15.328525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:15.328330    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc55ea3695c7de0cdc24650aba236df3838c35419493aba423f99e8741da733"
Oct 30 20:07:18.062233 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:18.062195    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"]
Oct 30 20:07:18.113044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:18.113013    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f666521b-xx7d7"]
Oct 30 20:07:19.621953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.621919    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"]
Oct 30 20:07:19.622560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.622544    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e312f4f5-febd-4877-b571-8e4c8000699d" containerName="revoke-keys"
Oct 30 20:07:19.622612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.622561    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e312f4f5-febd-4877-b571-8e4c8000699d" containerName="revoke-keys"
Oct 30 20:07:19.622659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.622649    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e312f4f5-febd-4877-b571-8e4c8000699d" containerName="revoke-keys"
Oct 30 20:07:19.627396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.627380    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.681508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.681482    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:19.686210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.686178    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"]
Oct 30 20:07:19.730377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.730351    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:19.730786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.730737    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:19.731461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.731434    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnn7\" (UniqueName: \"kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.731554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.731487    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.731703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.731679    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.731902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.731884    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:19.832526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.832491    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.832725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.832656    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnn7\" (UniqueName: \"kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.832725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.832686    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.833259 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.833238    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.835054 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.835031    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.894794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.894704    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnn7\" (UniqueName: \"kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7\") pod \"maas-api-revoke-sub-be70943b-pwbhp\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:19.941798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:19.941044    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:20.016029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:20.015983    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b33878-846c-433e-8263-91d56276a0bd" path="/var/lib/kubelet/pods/19b33878-846c-433e-8263-91d56276a0bd/volumes"
Oct 30 20:07:20.369777 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:20.369746    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"]
Oct 30 20:07:20.371467 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:20.371438    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e35c54c_1810_4fcf_ab42_28201a747b07.slice/crio-73c4ede2750105896c9e818c9c19d317a9febf50698885376d44ca2aaa753496 WatchSource:0}: Error finding container 73c4ede2750105896c9e818c9c19d317a9febf50698885376d44ca2aaa753496: Status 404 returned error can't find the container with id 73c4ede2750105896c9e818c9c19d317a9febf50698885376d44ca2aaa753496
Oct 30 20:07:21.421050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:21.421014    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp" event={"ID":"4e35c54c-1810-4fcf-ab42-28201a747b07","Type":"ContainerStarted","Data":"7bca86fa8f8da9b17c2b08927525ba70de6f76fb1cb5f2c82bb235127fa42436"}
Oct 30 20:07:21.421640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:21.421614    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp" event={"ID":"4e35c54c-1810-4fcf-ab42-28201a747b07","Type":"ContainerStarted","Data":"73c4ede2750105896c9e818c9c19d317a9febf50698885376d44ca2aaa753496"}
Oct 30 20:07:21.551836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:21.548264    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp" podStartSLOduration=2.548224029 podStartE2EDuration="2.548224029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:21.546345871 +0000 UTC m=+1537.710747514" watchObservedRunningTime="2026-10-30 20:07:21.548224029 +0000 UTC m=+1537.712625672"
Oct 30 20:07:22.486811 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.486758    4917 generic.go:356] "Generic (PLEG): container finished" podID="4e35c54c-1810-4fcf-ab42-28201a747b07" containerID="7bca86fa8f8da9b17c2b08927525ba70de6f76fb1cb5f2c82bb235127fa42436" exitCode=0
Oct 30 20:07:22.487475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.487446    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp" event={"ID":"4e35c54c-1810-4fcf-ab42-28201a747b07","Type":"ContainerDied","Data":"7bca86fa8f8da9b17c2b08927525ba70de6f76fb1cb5f2c82bb235127fa42436"}
Oct 30 20:07:22.732207 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.731040    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.732504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.732474    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:07:22.738838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.734838    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.892293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.892236    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:07:22.924364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:22.924322    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.088987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.088875    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.150795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.150588    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.158794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.151030    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.274050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.273889    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:23.274050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.273974    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.311687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.310867    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.699278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.699224    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.839448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.839404    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.893911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.893853    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.894587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.894564    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:07:23.954962 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.940854    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.954962 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.953847    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:07:23.974229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.974200    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:07:23.990881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:23.990436    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:07:24.202015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.200613    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.202015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.200985    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:07:24.365788 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.362419    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:07:24.422569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.422540    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:24.572289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.572181    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca\") pod \"4e35c54c-1810-4fcf-ab42-28201a747b07\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") "
Oct 30 20:07:24.572460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.572411    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca\") pod \"4e35c54c-1810-4fcf-ab42-28201a747b07\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") "
Oct 30 20:07:24.572919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.572882    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca" pod "4e35c54c-1810-4fcf-ab42-28201a747b07" (UID: "4e35c54c-1810-4fcf-ab42-28201a747b07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:24.573058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.572952    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7\" (UniqueName: \"kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7\") pod \"4e35c54c-1810-4fcf-ab42-28201a747b07\" (UID: \"4e35c54c-1810-4fcf-ab42-28201a747b07\") "
Oct 30 20:07:24.573691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.573667    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:24.577412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.576124    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca" pod "4e35c54c-1810-4fcf-ab42-28201a747b07" (UID: "4e35c54c-1810-4fcf-ab42-28201a747b07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:24.577659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.577630    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7" pod "4e35c54c-1810-4fcf-ab42-28201a747b07" (UID: "4e35c54c-1810-4fcf-ab42-28201a747b07"). InnerVolumeSpecName "kube-api-access-hjnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:24.675485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.675446    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e35c54c-1810-4fcf-ab42-28201a747b07-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:24.675485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.675488    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjnn7\" (UniqueName: \"kubernetes.io/projected/4e35c54c-1810-4fcf-ab42-28201a747b07-kube-api-access-hjnn7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:24.805575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.805541    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"
Oct 30 20:07:24.805575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.805559    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp" event={"ID":"4e35c54c-1810-4fcf-ab42-28201a747b07","Type":"ContainerDied","Data":"73c4ede2750105896c9e818c9c19d317a9febf50698885376d44ca2aaa753496"}
Oct 30 20:07:24.806086 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:24.805614    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c4ede2750105896c9e818c9c19d317a9febf50698885376d44ca2aaa753496"
Oct 30 20:07:25.060383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:25.060351    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"]
Oct 30 20:07:25.064444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:25.064415    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d858750-pgxb8"]
Oct 30 20:07:26.004158 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:26.004129    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ceb1d2-8b85-41de-a302-2e5abe7a467a" path="/var/lib/kubelet/pods/17ceb1d2-8b85-41de-a302-2e5abe7a467a/volumes"
Oct 30 20:07:30.057570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.057537    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"]
Oct 30 20:07:30.108590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.108549    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce5cdc-pnswb"]
Oct 30 20:07:30.293327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.293288    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"]
Oct 30 20:07:30.293970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.293951    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4e35c54c-1810-4fcf-ab42-28201a747b07" containerName="revoke-keys"
Oct 30 20:07:30.293970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.293969    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e35c54c-1810-4fcf-ab42-28201a747b07" containerName="revoke-keys"
Oct 30 20:07:30.294113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.294062    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e35c54c-1810-4fcf-ab42-28201a747b07" containerName="revoke-keys"
Oct 30 20:07:30.299028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.299004    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.371256 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.371217    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"]
Oct 30 20:07:30.372057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.372038    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:30.372136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.372123    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:30.372241 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.372219    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:30.372382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.372333    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:30.455110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.455075    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wrt\" (UniqueName: \"kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.455278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.455192    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.455278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.455224    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.557027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.556992    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wrt\" (UniqueName: \"kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.557185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.557075    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.557185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.557104    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.557808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.557740    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.559476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.559455    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.619365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.619288    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wrt\" (UniqueName: \"kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt\") pod \"maas-api-revoke-sub-ff7be398-76rb7\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:30.621120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:30.621096    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:31.004752 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:31.004718    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfba6e83_3211_4870_b9f4_74aebac007a2.slice/crio-fff600b3d14ddc8011c61b002d9db6e97bbad35c02404918e84590ae4240bcb1 WatchSource:0}: Error finding container fff600b3d14ddc8011c61b002d9db6e97bbad35c02404918e84590ae4240bcb1: Status 404 returned error can't find the container with id fff600b3d14ddc8011c61b002d9db6e97bbad35c02404918e84590ae4240bcb1
Oct 30 20:07:31.005038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:31.005016    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"]
Oct 30 20:07:31.865701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:31.865670    4917 generic.go:356] "Generic (PLEG): container finished" podID="dfba6e83-3211-4870-b9f4-74aebac007a2" containerID="a6f11a06c96c3680e1578859cd30f4a10b21e232ee8bb2bc19668a66bbf5b883" exitCode=0
Oct 30 20:07:31.866064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:31.865748    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7" event={"ID":"dfba6e83-3211-4870-b9f4-74aebac007a2","Type":"ContainerDied","Data":"a6f11a06c96c3680e1578859cd30f4a10b21e232ee8bb2bc19668a66bbf5b883"}
Oct 30 20:07:31.866064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:31.865820    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7" event={"ID":"dfba6e83-3211-4870-b9f4-74aebac007a2","Type":"ContainerStarted","Data":"fff600b3d14ddc8011c61b002d9db6e97bbad35c02404918e84590ae4240bcb1"}
Oct 30 20:07:32.004950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.004920    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0" path="/var/lib/kubelet/pods/4c04c20c-1eb6-4b9a-8378-45c2b2fa14e0/volumes"
Oct 30 20:07:32.308933 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.308897    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"]
Oct 30 20:07:32.313151 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.313133    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.369764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.367797    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-407632\"/\"distinct-2-w5-407632-kserve-self-signed-certs\""
Oct 30 20:07:32.422560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.422529    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"]
Oct 30 20:07:32.484131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.484096    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.484436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.484151    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.484436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.484240    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmtj5\" (UniqueName: \"kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.484436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.484329    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.484436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.484371    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.484436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.484403    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.585639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.585538    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.585639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.585591    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.585639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.585619    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmtj5\" (UniqueName: \"kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.586025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.585659    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.586025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.585689    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.586025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.585716    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.586183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.586020    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.586183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.586063    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.586357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.586320    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.588055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.588034    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.588484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.588469    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.648071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.648040    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmtj5\" (UniqueName: \"kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5\") pod \"distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") " pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:32.927303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:32.927265    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:33.046892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.046863    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:33.195975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.195929    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca\") pod \"dfba6e83-3211-4870-b9f4-74aebac007a2\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") "
Oct 30 20:07:33.196204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.196166    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca\") pod \"dfba6e83-3211-4870-b9f4-74aebac007a2\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") "
Oct 30 20:07:33.196302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.196221    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt\" (UniqueName: \"kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt\") pod \"dfba6e83-3211-4870-b9f4-74aebac007a2\" (UID: \"dfba6e83-3211-4870-b9f4-74aebac007a2\") "
Oct 30 20:07:33.196302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.196270    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca" pod "dfba6e83-3211-4870-b9f4-74aebac007a2" (UID: "dfba6e83-3211-4870-b9f4-74aebac007a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:33.197138 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.197108    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:33.198522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.198484    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt" pod "dfba6e83-3211-4870-b9f4-74aebac007a2" (UID: "dfba6e83-3211-4870-b9f4-74aebac007a2"). InnerVolumeSpecName "kube-api-access-68wrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:33.198959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.198938    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca" pod "dfba6e83-3211-4870-b9f4-74aebac007a2" (UID: "dfba6e83-3211-4870-b9f4-74aebac007a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:33.298249 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.298212    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfba6e83-3211-4870-b9f4-74aebac007a2-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:33.298249 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.298242    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wrt\" (UniqueName: \"kubernetes.io/projected/dfba6e83-3211-4870-b9f4-74aebac007a2-kube-api-access-68wrt\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:33.353605 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:33.353575    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod704961c4_062f_4508_9c50_d37652ea729d.slice/crio-768195a573dd68b1cee9b467f0c98c4198e4751121d480779a012f674426a604 WatchSource:0}: Error finding container 768195a573dd68b1cee9b467f0c98c4198e4751121d480779a012f674426a604: Status 404 returned error can't find the container with id 768195a573dd68b1cee9b467f0c98c4198e4751121d480779a012f674426a604
Oct 30 20:07:33.354456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.354430    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"]
Oct 30 20:07:33.883389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.883291    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" event={"ID":"704961c4-062f-4508-9c50-d37652ea729d","Type":"ContainerStarted","Data":"a5b44da9faedcaf6dd80273ea0320d39246b4b986fc99b0216ba8dd3128167de"}
Oct 30 20:07:33.883389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.883336    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" event={"ID":"704961c4-062f-4508-9c50-d37652ea729d","Type":"ContainerStarted","Data":"768195a573dd68b1cee9b467f0c98c4198e4751121d480779a012f674426a604"}
Oct 30 20:07:33.884891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.884867    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"
Oct 30 20:07:33.885022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.884913    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7" event={"ID":"dfba6e83-3211-4870-b9f4-74aebac007a2","Type":"ContainerDied","Data":"fff600b3d14ddc8011c61b002d9db6e97bbad35c02404918e84590ae4240bcb1"}
Oct 30 20:07:33.885022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.884993    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff600b3d14ddc8011c61b002d9db6e97bbad35c02404918e84590ae4240bcb1"
Oct 30 20:07:33.928507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:33.928466    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:34.005969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:34.005908    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podStartSLOduration=2.005890598 podStartE2EDuration="2.005890598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:34.003036735 +0000 UTC m=+1550.167438402" watchObservedRunningTime="2026-10-30 20:07:34.005890598 +0000 UTC m=+1550.170292241"
Oct 30 20:07:34.062233 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:34.062202    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"]
Oct 30 20:07:34.066479 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:34.066457    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6763c652-fgqbc"]
Oct 30 20:07:36.004613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:36.004583    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18d0377-d700-40e1-a8ea-12c63f50f64c" path="/var/lib/kubelet/pods/c18d0377-d700-40e1-a8ea-12c63f50f64c/volumes"
Oct 30 20:07:39.480652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.480619    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"]
Oct 30 20:07:39.481236 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.481220    4917 cpu_manager.go:414] "containerMap: removing container" podUID="dfba6e83-3211-4870-b9f4-74aebac007a2" containerName="revoke-keys"
Oct 30 20:07:39.481275 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.481237    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfba6e83-3211-4870-b9f4-74aebac007a2" containerName="revoke-keys"
Oct 30 20:07:39.481362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.481353    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfba6e83-3211-4870-b9f4-74aebac007a2" containerName="revoke-keys"
Oct 30 20:07:39.486238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.486214    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.543728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.543697    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"]
Oct 30 20:07:39.572379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.572346    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.572545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.572394    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.572545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.572415    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjr6v\" (UniqueName: \"kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.588484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.588456    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:39.588624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.588500    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:39.590443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.590427    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:39.590564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.590550    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:39.674527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.674498    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.675089 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.674543    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.675089 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.674565    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjr6v\" (UniqueName: \"kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.675416 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.675395    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.676883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.676858    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.682699 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:07:39.682673    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:07:39.736922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.736842    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjr6v\" (UniqueName: \"kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v\") pod \"maas-api-revoke-sub-5485ca1b-zfnvc\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:39.799830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:39.799798    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:40.183293 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:40.183262    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d12d49f_ed02_4b5e_9045_abb8841d2c78.slice/crio-7091b8ecba08dad17f67c805744d85c1348a9b3306fb6e48e91eb7a3e09e1faa WatchSource:0}: Error finding container 7091b8ecba08dad17f67c805744d85c1348a9b3306fb6e48e91eb7a3e09e1faa: Status 404 returned error can't find the container with id 7091b8ecba08dad17f67c805744d85c1348a9b3306fb6e48e91eb7a3e09e1faa
Oct 30 20:07:40.183410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:40.183292    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"]
Oct 30 20:07:40.942689 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:40.942650    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc" event={"ID":"6d12d49f-ed02-4b5e-9045-abb8841d2c78","Type":"ContainerStarted","Data":"bca897581be9b0b246faf25cc11cd092defd0d59272b12946f6bb19248e02966"}
Oct 30 20:07:40.943196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:40.942737    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc" event={"ID":"6d12d49f-ed02-4b5e-9045-abb8841d2c78","Type":"ContainerStarted","Data":"7091b8ecba08dad17f67c805744d85c1348a9b3306fb6e48e91eb7a3e09e1faa"}
Oct 30 20:07:41.066229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:41.066173    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc" podStartSLOduration=2.06615842 podStartE2EDuration="2.06615842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:41.063011967 +0000 UTC m=+1557.227413610" watchObservedRunningTime="2026-10-30 20:07:41.06615842 +0000 UTC m=+1557.230560062"
Oct 30 20:07:41.953159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:41.953127    4917 generic.go:356] "Generic (PLEG): container finished" podID="6d12d49f-ed02-4b5e-9045-abb8841d2c78" containerID="bca897581be9b0b246faf25cc11cd092defd0d59272b12946f6bb19248e02966" exitCode=0
Oct 30 20:07:41.953617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:41.953221    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc" event={"ID":"6d12d49f-ed02-4b5e-9045-abb8841d2c78","Type":"ContainerDied","Data":"bca897581be9b0b246faf25cc11cd092defd0d59272b12946f6bb19248e02966"}
Oct 30 20:07:42.928345 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:42.928310    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:42.941657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:42.941631    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:42.973067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:42.973042    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:43.127834 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.127808    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:43.223178 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.223095    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca\") pod \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") "
Oct 30 20:07:43.223500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.223181    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v\" (UniqueName: \"kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v\") pod \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") "
Oct 30 20:07:43.223500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.223302    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca\") pod \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\" (UID: \"6d12d49f-ed02-4b5e-9045-abb8841d2c78\") "
Oct 30 20:07:43.223902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.223875    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca" pod "6d12d49f-ed02-4b5e-9045-abb8841d2c78" (UID: "6d12d49f-ed02-4b5e-9045-abb8841d2c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:43.225688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.225662    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v" pod "6d12d49f-ed02-4b5e-9045-abb8841d2c78" (UID: "6d12d49f-ed02-4b5e-9045-abb8841d2c78"). InnerVolumeSpecName "kube-api-access-mjr6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:43.225814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.225695    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca" pod "6d12d49f-ed02-4b5e-9045-abb8841d2c78" (UID: "6d12d49f-ed02-4b5e-9045-abb8841d2c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:43.325026 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.324991    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:43.325026 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.325020    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjr6v\" (UniqueName: \"kubernetes.io/projected/6d12d49f-ed02-4b5e-9045-abb8841d2c78-kube-api-access-mjr6v\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:43.325026 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.325030    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d12d49f-ed02-4b5e-9045-abb8841d2c78-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:43.970152 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.970126    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"
Oct 30 20:07:43.970351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.970167    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc" event={"ID":"6d12d49f-ed02-4b5e-9045-abb8841d2c78","Type":"ContainerDied","Data":"7091b8ecba08dad17f67c805744d85c1348a9b3306fb6e48e91eb7a3e09e1faa"}
Oct 30 20:07:43.970351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:43.970215    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7091b8ecba08dad17f67c805744d85c1348a9b3306fb6e48e91eb7a3e09e1faa"
Oct 30 20:07:44.210085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:44.210054    4917 scope.go:122] "RemoveContainer" containerID="8a9e3e3f75e6b5b5557ce37fcf0ce3ebc8d9996ee16729e6dc3e61e09cd94ccd"
Oct 30 20:07:44.220455 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:44.220433    4917 scope.go:122] "RemoveContainer" containerID="b492e6f8ff3debe7bef119626d90f2e354da518d4902db353b5fcce2a29bf01e"
Oct 30 20:07:44.229983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:44.229964    4917 scope.go:122] "RemoveContainer" containerID="df7af9bb40ddedda2e6a0f6d1d9e117796c2609af02fb8d7876b466320ee6510"
Oct 30 20:07:46.991917 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:46.991884    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"]
Oct 30 20:07:46.992946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:46.992925    4917 cpu_manager.go:414] "containerMap: removing container" podUID="6d12d49f-ed02-4b5e-9045-abb8841d2c78" containerName="revoke-keys"
Oct 30 20:07:46.993001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:46.992949    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d12d49f-ed02-4b5e-9045-abb8841d2c78" containerName="revoke-keys"
Oct 30 20:07:46.993124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:46.993112    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d12d49f-ed02-4b5e-9045-abb8841d2c78" containerName="revoke-keys"
Oct 30 20:07:46.999242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:46.999214    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.063905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.063874    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"]
Oct 30 20:07:47.070075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.070042    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsd7\" (UniqueName: \"kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.070215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.070150    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.070814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.070758    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.106941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.106907    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:47.107068 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.106959    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:47.107068 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.107001    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:47.108072 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.108054    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:47.172107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.172074    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsd7\" (UniqueName: \"kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.172267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.172129    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.172267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.172209    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.172926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.172906    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.174522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.174504    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.218185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.218158    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"]
Oct 30 20:07:47.222493 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.222475    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.242484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.242431    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsd7\" (UniqueName: \"kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7\") pod \"maas-api-revoke-sub-43a1bb42-8jf9f\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.273893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.273843    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274007 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.273942    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274014    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274121 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274094    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274308    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274368    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274442 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274425    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrsw\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274574    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.274685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.274668    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.308914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.308883    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"]
Oct 30 20:07:47.313633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.313603    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:47.375925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.375885    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376091 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.375953    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376091 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.375982    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrsw\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376091    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376154    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376203    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376209    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376263    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376308    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376336    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376675    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376764    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.376893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.376854    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.377402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.377380    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.378347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.378328    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.379163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.379139    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.449480 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.449446    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrsw\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.462577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.462548    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.537467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.537383    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:47.725064 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:47.725035    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70646b36_d40d_4fbf_b398_9d4a3378fe3e.slice/crio-cb03c1f2cf178dd6a467f6029a91c2baeca9a6fbdb6132896955bf50d6cb3145 WatchSource:0}: Error finding container cb03c1f2cf178dd6a467f6029a91c2baeca9a6fbdb6132896955bf50d6cb3145: Status 404 returned error can't find the container with id cb03c1f2cf178dd6a467f6029a91c2baeca9a6fbdb6132896955bf50d6cb3145
Oct 30 20:07:47.726080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.726061    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"]
Oct 30 20:07:47.924665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.924638    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"]
Oct 30 20:07:47.925053 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:47.925029    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408c02f5_9912_4a99_b957_b07c20128df0.slice/crio-a51f2cc9d0ca55b9506a950347c69064cf799f632f8f8d8aedad93ee30706e93 WatchSource:0}: Error finding container a51f2cc9d0ca55b9506a950347c69064cf799f632f8f8d8aedad93ee30706e93: Status 404 returned error can't find the container with id a51f2cc9d0ca55b9506a950347c69064cf799f632f8f8d8aedad93ee30706e93
Oct 30 20:07:47.927734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.927698    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:47.927837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.927765    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:47.927908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:47.927848    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:07:48.006786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.006741    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f" event={"ID":"70646b36-d40d-4fbf-b398-9d4a3378fe3e","Type":"ContainerStarted","Data":"cb03c1f2cf178dd6a467f6029a91c2baeca9a6fbdb6132896955bf50d6cb3145"}
Oct 30 20:07:48.007203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.007088    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" event={"ID":"408c02f5-9912-4a99-b957-b07c20128df0","Type":"ContainerStarted","Data":"d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89"}
Oct 30 20:07:48.007203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.007135    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" event={"ID":"408c02f5-9912-4a99-b957-b07c20128df0","Type":"ContainerStarted","Data":"a51f2cc9d0ca55b9506a950347c69064cf799f632f8f8d8aedad93ee30706e93"}
Oct 30 20:07:48.130865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.130806    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podStartSLOduration=1.130790017 podStartE2EDuration="1.130790017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:48.129132009 +0000 UTC m=+1564.293533688" watchObservedRunningTime="2026-10-30 20:07:48.130790017 +0000 UTC m=+1564.295191662"
Oct 30 20:07:48.538012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.537980    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:48.539698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.539676    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:48.539797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:48.539724    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:49.016594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:49.016560    4917 generic.go:356] "Generic (PLEG): container finished" podID="70646b36-d40d-4fbf-b398-9d4a3378fe3e" containerID="892a619f665b2b657fbb037c9c293208961bc71eb05187b1f19077451b6db034" exitCode=0
Oct 30 20:07:49.017015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:49.016639    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f" event={"ID":"70646b36-d40d-4fbf-b398-9d4a3378fe3e","Type":"ContainerDied","Data":"892a619f665b2b657fbb037c9c293208961bc71eb05187b1f19077451b6db034"}
Oct 30 20:07:49.538657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:49.538623    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:49.538863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:49.538676    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:50.272183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.272159    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:50.314884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.314802    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7\" (UniqueName: \"kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7\") pod \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") "
Oct 30 20:07:50.315009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.314886    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca\") pod \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") "
Oct 30 20:07:50.315009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.314984    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca\") pod \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\" (UID: \"70646b36-d40d-4fbf-b398-9d4a3378fe3e\") "
Oct 30 20:07:50.315277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.315238    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca" pod "70646b36-d40d-4fbf-b398-9d4a3378fe3e" (UID: "70646b36-d40d-4fbf-b398-9d4a3378fe3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:50.315506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.315491    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:50.317065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.317042    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7" pod "70646b36-d40d-4fbf-b398-9d4a3378fe3e" (UID: "70646b36-d40d-4fbf-b398-9d4a3378fe3e"). InnerVolumeSpecName "kube-api-access-dqsd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:50.317157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.317118    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca" pod "70646b36-d40d-4fbf-b398-9d4a3378fe3e" (UID: "70646b36-d40d-4fbf-b398-9d4a3378fe3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:50.417797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.417734    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsd7\" (UniqueName: \"kubernetes.io/projected/70646b36-d40d-4fbf-b398-9d4a3378fe3e-kube-api-access-dqsd7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:50.417797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.417784    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70646b36-d40d-4fbf-b398-9d4a3378fe3e-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:50.538257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.538216    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:50.538454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:50.538275    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:51.034743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.034699    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f" event={"ID":"70646b36-d40d-4fbf-b398-9d4a3378fe3e","Type":"ContainerDied","Data":"cb03c1f2cf178dd6a467f6029a91c2baeca9a6fbdb6132896955bf50d6cb3145"}
Oct 30 20:07:51.034957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.034819    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb03c1f2cf178dd6a467f6029a91c2baeca9a6fbdb6132896955bf50d6cb3145"
Oct 30 20:07:51.034957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.034924    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"
Oct 30 20:07:51.078236 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.078204    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"]
Oct 30 20:07:51.082428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.082405    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d10a8f0-ltx5c"]
Oct 30 20:07:51.538351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.538316    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:51.538743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:51.538384    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:52.008369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:52.007891    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf63ed7a-4335-4e6e-952c-75fa436da582" path="/var/lib/kubelet/pods/cf63ed7a-4335-4e6e-952c-75fa436da582/volumes"
Oct 30 20:07:52.537918 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:52.537886    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:52.538174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:52.537936    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:53.538733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:53.538699    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:53.539220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:53.538754    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:54.538601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.538562    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:07:54.538828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.538617    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:07:54.810142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.810057    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"]
Oct 30 20:07:54.810658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.810641    4917 cpu_manager.go:414] "containerMap: removing container" podUID="70646b36-d40d-4fbf-b398-9d4a3378fe3e" containerName="revoke-keys"
Oct 30 20:07:54.810658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.810658    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70646b36-d40d-4fbf-b398-9d4a3378fe3e" containerName="revoke-keys"
Oct 30 20:07:54.810866 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.810834    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="70646b36-d40d-4fbf-b398-9d4a3378fe3e" containerName="revoke-keys"
Oct 30 20:07:54.815426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.815406    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:54.874264 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.874226    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"]
Oct 30 20:07:54.898585 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.898545    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:54.898748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.898609    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:54.898748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.898718    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fsl\" (UniqueName: \"kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:54.918958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.918929    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:54.918958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.918949    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:07:54.919145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.918931    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:54.919145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:54.919033    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:07:55.000103 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.000074    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.000312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.000133    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.000391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.000301    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fsl\" (UniqueName: \"kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.000820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.000796    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.002624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.002595    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.062079 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.062012    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fsl\" (UniqueName: \"kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl\") pod \"maas-api-revoke-sub-e2048026-46ssp\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.128905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.128870    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:55.130590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.130554    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:07:55.539919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.539887    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"]
Oct 30 20:07:55.540910 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:07:55.540871    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9abb9c_4295_4440_afa6_ec70680f6e65.slice/crio-1490a2cd8877e190f429e34ffdac6ebf5ae2cbba1b105567bd2cb2f7b6830b53 WatchSource:0}: Error finding container 1490a2cd8877e190f429e34ffdac6ebf5ae2cbba1b105567bd2cb2f7b6830b53: Status 404 returned error can't find the container with id 1490a2cd8877e190f429e34ffdac6ebf5ae2cbba1b105567bd2cb2f7b6830b53
Oct 30 20:07:55.596919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.596885    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:55.597679 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.597654    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:55.598381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:55.598366    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:07:56.105934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:56.105899    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp" event={"ID":"df9abb9c-4295-4440-afa6-ec70680f6e65","Type":"ContainerStarted","Data":"c12810d8fc97030dd127ad71e2aa56abadbc67f5c84bda125fa40d5ea18d983a"}
Oct 30 20:07:56.106382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:56.106353    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp" event={"ID":"df9abb9c-4295-4440-afa6-ec70680f6e65","Type":"ContainerStarted","Data":"1490a2cd8877e190f429e34ffdac6ebf5ae2cbba1b105567bd2cb2f7b6830b53"}
Oct 30 20:07:56.254793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:56.247868    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp" podStartSLOduration=2.247845985 podStartE2EDuration="2.247845985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:56.245683497 +0000 UTC m=+1572.410085139" watchObservedRunningTime="2026-10-30 20:07:56.247845985 +0000 UTC m=+1572.412247627"
Oct 30 20:07:57.135853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.135400    4917 generic.go:356] "Generic (PLEG): container finished" podID="df9abb9c-4295-4440-afa6-ec70680f6e65" containerID="c12810d8fc97030dd127ad71e2aa56abadbc67f5c84bda125fa40d5ea18d983a" exitCode=0
Oct 30 20:07:57.140793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.137841    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp" event={"ID":"df9abb9c-4295-4440-afa6-ec70680f6e65","Type":"ContainerDied","Data":"c12810d8fc97030dd127ad71e2aa56abadbc67f5c84bda125fa40d5ea18d983a"}
Oct 30 20:07:57.801794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.796889    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.801794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.798851    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.825795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.808886    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.833791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.831894    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.884794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.883890    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.884794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.884749    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.973795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.972855    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.978795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.977843    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.996792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:57.992891    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.031794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.029551    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.078360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.077888    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.247115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.245885    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.331750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.330874    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.351652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.350867    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.646842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.641860    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vvxc" podUID="16655b4a-806f-43b4-812a-9782b3237211" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.28:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.852796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.800902    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.852796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.801311    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.852796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.801658    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:07:58.852796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.801719    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.902182    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.936348    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.936701    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.937010    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.965337    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.965403    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.965441    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.965480    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.977402    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.977885    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.978953    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.991887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.979258    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:07:58.994255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.993129    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.994255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:58.993492    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:07:59.070843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.031493    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:07:59.070843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.043528    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:07:59.148076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.148045    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:07:59.170872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.170838    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:59.337851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.332248    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.337851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.332724    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:07:59.357067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.357026    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.357526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.357503    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:07:59.521837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.521058    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca\") pod \"df9abb9c-4295-4440-afa6-ec70680f6e65\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") "
Oct 30 20:07:59.521837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.521484    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl\" (UniqueName: \"kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl\") pod \"df9abb9c-4295-4440-afa6-ec70680f6e65\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") "
Oct 30 20:07:59.521837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.521585    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca\") pod \"df9abb9c-4295-4440-afa6-ec70680f6e65\" (UID: \"df9abb9c-4295-4440-afa6-ec70680f6e65\") "
Oct 30 20:07:59.527496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.527462    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:07:59.569829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.569788    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca" pod "df9abb9c-4295-4440-afa6-ec70680f6e65" (UID: "df9abb9c-4295-4440-afa6-ec70680f6e65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:59.603957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.583864    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca" pod "df9abb9c-4295-4440-afa6-ec70680f6e65" (UID: "df9abb9c-4295-4440-afa6-ec70680f6e65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:59.685814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.683805    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:07:59.706107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.706075    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:59.706358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.706337    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9abb9c-4295-4440-afa6-ec70680f6e65-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:59.715711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.715672    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.719010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.718986    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:07:59.726197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.719763    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:07:59.726340 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.726306    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl" pod "df9abb9c-4295-4440-afa6-ec70680f6e65" (UID: "df9abb9c-4295-4440-afa6-ec70680f6e65"). InnerVolumeSpecName "kube-api-access-q9fsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:59.747092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.746397    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:07:59.767064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.760690    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:07:59.767064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.761561    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:07:59.769603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.767856    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:07:59.769603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.767912    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:07:59.769603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.767942    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:07:59.769603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.768540    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:07:59.771573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.771551    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"
Oct 30 20:07:59.799175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.772706    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp" event={"ID":"df9abb9c-4295-4440-afa6-ec70680f6e65","Type":"ContainerDied","Data":"1490a2cd8877e190f429e34ffdac6ebf5ae2cbba1b105567bd2cb2f7b6830b53"}
Oct 30 20:07:59.799175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.798008    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1490a2cd8877e190f429e34ffdac6ebf5ae2cbba1b105567bd2cb2f7b6830b53"
Oct 30 20:07:59.808785 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:07:59.808484    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9fsl\" (UniqueName: \"kubernetes.io/projected/df9abb9c-4295-4440-afa6-ec70680f6e65-kube-api-access-q9fsl\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:59.865804 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:07:59.865709    4917 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9abb9c_4295_4440_afa6_ec70680f6e65.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:08:01.065122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:01.065083    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"]
Oct 30 20:08:01.069734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:01.069706    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17599f98-t4nsz"]
Oct 30 20:08:02.007045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.007012    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a5512d-1d26-4a2d-af4f-b55c0483a619" path="/var/lib/kubelet/pods/91a5512d-1d26-4a2d-af4f-b55c0483a619/volumes"
Oct 30 20:08:02.293125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.293038    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"]
Oct 30 20:08:02.293663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.293649    4917 cpu_manager.go:414] "containerMap: removing container" podUID="df9abb9c-4295-4440-afa6-ec70680f6e65" containerName="revoke-keys"
Oct 30 20:08:02.293719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.293664    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9abb9c-4295-4440-afa6-ec70680f6e65" containerName="revoke-keys"
Oct 30 20:08:02.293784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.293762    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9abb9c-4295-4440-afa6-ec70680f6e65" containerName="revoke-keys"
Oct 30 20:08:02.298553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.298531    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.360320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.360290    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"]
Oct 30 20:08:02.444076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444046    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444155    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444256 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444235    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444316 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444259    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkt8h\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444359    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444400    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444473    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444533    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.444644 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.444628    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545358    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545434    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545496    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545517    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkt8h\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545536    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545560    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545613    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545635    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.545733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545720    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.546135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545830    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.546135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.545981    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.546135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.546109    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.546292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.546179    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.546438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.546410    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.547969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.547950    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.548180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.548163    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.613792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.613746    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.614090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.614064    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkt8h\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.912331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.912235    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:02.941651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.941622    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"]
Oct 30 20:08:02.946350 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.946327    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:02.949638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.949608    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:02.949763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.949651    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:02.949987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:02.949962    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.001398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.001369    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:03.001569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.001551    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:03.001663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.001650    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:08:03.001730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.001718    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:08:03.003713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.003686    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"]
Oct 30 20:08:03.051135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.051102    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.051509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.051163    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.051509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.051283    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.052124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.052097    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.053744 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.053722    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.114107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.114076    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7\") pod \"maas-api-revoke-sub-8752d712-8btqj\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.284050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.284009    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:03.371666 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:03.371634    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda705b05f_e0b3_4b7c_bc09_693284568acc.slice/crio-ee90e6a319403921ab8bbc7ba73e9afa02ff904d36ed28f21d7582fc388bc362 WatchSource:0}: Error finding container ee90e6a319403921ab8bbc7ba73e9afa02ff904d36ed28f21d7582fc388bc362: Status 404 returned error can't find the container with id ee90e6a319403921ab8bbc7ba73e9afa02ff904d36ed28f21d7582fc388bc362
Oct 30 20:08:03.373246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.373214    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"]
Oct 30 20:08:03.374608 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.374576    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:03.374699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.374638    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:03.374699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.374669    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:03.671887 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:03.671855    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa12c27_3ab6_4fc5_911d_3b1a82a2f7e9.slice/crio-3649f2612c7446df50d60e81d42a8d655792c213100399bc5d13f11e1dc6eb96 WatchSource:0}: Error finding container 3649f2612c7446df50d60e81d42a8d655792c213100399bc5d13f11e1dc6eb96: Status 404 returned error can't find the container with id 3649f2612c7446df50d60e81d42a8d655792c213100399bc5d13f11e1dc6eb96
Oct 30 20:08:03.672718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.672691    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"]
Oct 30 20:08:03.807543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.807504    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" event={"ID":"a705b05f-e0b3-4b7c-bc09-693284568acc","Type":"ContainerStarted","Data":"868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb"}
Oct 30 20:08:03.807738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.807578    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" event={"ID":"a705b05f-e0b3-4b7c-bc09-693284568acc","Type":"ContainerStarted","Data":"ee90e6a319403921ab8bbc7ba73e9afa02ff904d36ed28f21d7582fc388bc362"}
Oct 30 20:08:03.809051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.809026    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj" event={"ID":"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9","Type":"ContainerStarted","Data":"3649f2612c7446df50d60e81d42a8d655792c213100399bc5d13f11e1dc6eb96"}
Oct 30 20:08:03.913224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.913196    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:03.932041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:03.931986    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podStartSLOduration=1.931972485 podStartE2EDuration="1.931972485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:03.930384482 +0000 UTC m=+1580.094786125" watchObservedRunningTime="2026-10-30 20:08:03.931972485 +0000 UTC m=+1580.096374126"
Oct 30 20:08:04.818642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:04.818562    4917 generic.go:356] "Generic (PLEG): container finished" podID="3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" containerID="d4e7bc07ff58e62bd757db327a2caa3e0e956fc3b982288b854e71ba243b05b5" exitCode=0
Oct 30 20:08:04.819027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:04.818644    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj" event={"ID":"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9","Type":"ContainerDied","Data":"d4e7bc07ff58e62bd757db327a2caa3e0e956fc3b982288b854e71ba243b05b5"}
Oct 30 20:08:04.914556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:04.914520    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:08:04.914696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:04.914580    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:08:05.913502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:05.913469    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:08:05.913972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:05.913533    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:08:06.018162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.018137    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:06.096194 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.096114    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7\") pod \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") "
Oct 30 20:08:06.096557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.096207    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca\") pod \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") "
Oct 30 20:08:06.096557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.096295    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca\") pod \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\" (UID: \"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9\") "
Oct 30 20:08:06.096557 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.096532    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca" pod "3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" (UID: "3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:06.096951 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.096897    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:06.098415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.098392    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7" pod "3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" (UID: "3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9"). InnerVolumeSpecName "kube-api-access-m9fn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:06.098521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.098479    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca" pod "3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" (UID: "3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:06.197841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.197805    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:06.197841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.197840    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9-kube-api-access-m9fn7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:06.835208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.835174    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"
Oct 30 20:08:06.835392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.835204    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj" event={"ID":"3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9","Type":"ContainerDied","Data":"3649f2612c7446df50d60e81d42a8d655792c213100399bc5d13f11e1dc6eb96"}
Oct 30 20:08:06.835392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.835286    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3649f2612c7446df50d60e81d42a8d655792c213100399bc5d13f11e1dc6eb96"
Oct 30 20:08:06.913533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.913501    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:08:06.913932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:06.913551    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:08:07.465795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.465449    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"]
Oct 30 20:08:07.467330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.467296    4917 cpu_manager.go:414] "containerMap: removing container" podUID="3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" containerName="revoke-keys"
Oct 30 20:08:07.467330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.467326    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" containerName="revoke-keys"
Oct 30 20:08:07.467507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.467481    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" containerName="revoke-keys"
Oct 30 20:08:07.471242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.471225    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.509266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.509236    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"]
Oct 30 20:08:07.516876 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.516837    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qs76\" (UniqueName: \"kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.517255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.517223    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.517341 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.517267    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.567796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.567584    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:07.567796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.567701    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:08:07.568016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.567850    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:07.575389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.575364    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:08:07.619322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.619290    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs76\" (UniqueName: \"kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.619574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.619551    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.619665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.619587    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.620354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.620336    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.622130 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.622088    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.683707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.683666    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs76\" (UniqueName: \"kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76\") pod \"maas-api-revoke-sub-b23660b5-lsv5q\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.784897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.784807    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:07.913649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.913604    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:08:07.914112 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:07.913671    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:08:08.178548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:08.178513    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"]
Oct 30 20:08:08.181187 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:08.181155    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd9f7b5_311e_4d99_b6db_f93d071fec2d.slice/crio-528a466f211f63f6804accfd1da718b18700aed296570ad69a6115cf34d32a21 WatchSource:0}: Error finding container 528a466f211f63f6804accfd1da718b18700aed296570ad69a6115cf34d32a21: Status 404 returned error can't find the container with id 528a466f211f63f6804accfd1da718b18700aed296570ad69a6115cf34d32a21
Oct 30 20:08:08.885894 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:08.885849    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q" event={"ID":"afd9f7b5-311e-4d99-b6db-f93d071fec2d","Type":"ContainerStarted","Data":"d868b6fde64b36d6ce156678d3b3cca323bc03b91b0645489eee817cbdbe77f0"}
Oct 30 20:08:08.886178 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:08.886157    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q" event={"ID":"afd9f7b5-311e-4d99-b6db-f93d071fec2d","Type":"ContainerStarted","Data":"528a466f211f63f6804accfd1da718b18700aed296570ad69a6115cf34d32a21"}
Oct 30 20:08:08.912923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:08.912892    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:08:08.913038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:08.912943    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:08:09.005349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:09.005290    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q" podStartSLOduration=2.005276503 podStartE2EDuration="2.005276503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:09.002851916 +0000 UTC m=+1585.167253559" watchObservedRunningTime="2026-10-30 20:08:09.005276503 +0000 UTC m=+1585.169678185"
Oct 30 20:08:09.689528 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:08:09.689479    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:08:09.895454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:09.895420    4917 generic.go:356] "Generic (PLEG): container finished" podID="afd9f7b5-311e-4d99-b6db-f93d071fec2d" containerID="d868b6fde64b36d6ce156678d3b3cca323bc03b91b0645489eee817cbdbe77f0" exitCode=0
Oct 30 20:08:09.895639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:09.895506    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q" event={"ID":"afd9f7b5-311e-4d99-b6db-f93d071fec2d","Type":"ContainerDied","Data":"d868b6fde64b36d6ce156678d3b3cca323bc03b91b0645489eee817cbdbe77f0"}
Oct 30 20:08:09.913469 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:09.913443    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:08:09.913600 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:09.913491    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:08:11.091828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.091798    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:11.102293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.102264    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"]
Oct 30 20:08:11.103221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.103201    4917 cpu_manager.go:414] "containerMap: removing container" podUID="afd9f7b5-311e-4d99-b6db-f93d071fec2d" containerName="revoke-keys"
Oct 30 20:08:11.103286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.103226    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd9f7b5-311e-4d99-b6db-f93d071fec2d" containerName="revoke-keys"
Oct 30 20:08:11.103452 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.103439    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd9f7b5-311e-4d99-b6db-f93d071fec2d" containerName="revoke-keys"
Oct 30 20:08:11.107327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.107306    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.189353 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189321    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76\" (UniqueName: \"kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76\") pod \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") "
Oct 30 20:08:11.189523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189380    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca\") pod \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") "
Oct 30 20:08:11.189523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189493    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca\") pod \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\" (UID: \"afd9f7b5-311e-4d99-b6db-f93d071fec2d\") "
Oct 30 20:08:11.189855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189834    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.189983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189875    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7x8d\" (UniqueName: \"kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.189983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189904    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.189983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.189923    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.190147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.190013    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.190147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.190089    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.190620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.190553    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca" pod "afd9f7b5-311e-4d99-b6db-f93d071fec2d" (UID: "afd9f7b5-311e-4d99-b6db-f93d071fec2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:11.192267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.192236    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca" pod "afd9f7b5-311e-4d99-b6db-f93d071fec2d" (UID: "afd9f7b5-311e-4d99-b6db-f93d071fec2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:11.192362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.192307    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76" pod "afd9f7b5-311e-4d99-b6db-f93d071fec2d" (UID: "afd9f7b5-311e-4d99-b6db-f93d071fec2d"). InnerVolumeSpecName "kube-api-access-5qs76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:11.196326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.196286    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"distinct-2-w2-d67ea9-kserve-self-signed-certs\""
Oct 30 20:08:11.204823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.204796    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"]
Oct 30 20:08:11.212160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.212139    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:11.212978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.212803    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:11.213462 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.213440    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:08:11.291883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.291847    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292079 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.291900    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7x8d\" (UniqueName: \"kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292079 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.292030    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.292079    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.292321    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.292490    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.292680    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.292839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.292809    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.293050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.293031    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qs76\" (UniqueName: \"kubernetes.io/projected/afd9f7b5-311e-4d99-b6db-f93d071fec2d-kube-api-access-5qs76\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:11.293129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.293056    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:11.293129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.293073    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd9f7b5-311e-4d99-b6db-f93d071fec2d-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:11.293351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.293330    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.295061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.295033    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.295469 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.295452    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.359780 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.359729    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7x8d\" (UniqueName: \"kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d\") pod \"distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.425313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.425280    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:11.868061 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:11.868028    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06528167_b423_4021_a6da_6fd10e1eab57.slice/crio-2ea68c701fc09c09af025fbcb2c01d91c9ad1782e802b5a65297371ac3cdb623 WatchSource:0}: Error finding container 2ea68c701fc09c09af025fbcb2c01d91c9ad1782e802b5a65297371ac3cdb623: Status 404 returned error can't find the container with id 2ea68c701fc09c09af025fbcb2c01d91c9ad1782e802b5a65297371ac3cdb623
Oct 30 20:08:11.869490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.869454    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"]
Oct 30 20:08:11.913537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.913517    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"
Oct 30 20:08:11.913658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.913526    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q" event={"ID":"afd9f7b5-311e-4d99-b6db-f93d071fec2d","Type":"ContainerDied","Data":"528a466f211f63f6804accfd1da718b18700aed296570ad69a6115cf34d32a21"}
Oct 30 20:08:11.913658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.913587    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="528a466f211f63f6804accfd1da718b18700aed296570ad69a6115cf34d32a21"
Oct 30 20:08:11.915119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:11.915094    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" event={"ID":"06528167-b423-4021-a6da-6fd10e1eab57","Type":"ContainerStarted","Data":"2ea68c701fc09c09af025fbcb2c01d91c9ad1782e802b5a65297371ac3cdb623"}
Oct 30 20:08:12.924485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:12.924445    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" event={"ID":"06528167-b423-4021-a6da-6fd10e1eab57","Type":"ContainerStarted","Data":"5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c"}
Oct 30 20:08:13.053544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:13.053492    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podStartSLOduration=2.053479443 podStartE2EDuration="2.053479443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:13.048642904 +0000 UTC m=+1589.213044547" watchObservedRunningTime="2026-10-30 20:08:13.053479443 +0000 UTC m=+1589.217881085"
Oct 30 20:08:13.425847 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:13.425808    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:14.923304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:14.923265    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"]
Oct 30 20:08:14.927797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:14.927762    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:14.988471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:14.988438    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"]
Oct 30 20:08:15.031572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.031530    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:08:15.031572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.031569    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:08:15.032524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.032482    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:15.032594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.032585    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:15.054435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.054408    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.054587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.054455    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.054669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.054656    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gqg\" (UniqueName: \"kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.156281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.156245    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.156455 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.156341    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.156535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.156490    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gqg\" (UniqueName: \"kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.157185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.157162    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.158684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.158665    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.220031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.219999    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gqg\" (UniqueName: \"kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg\") pod \"maas-api-revoke-sub-d12624a1-q8xmf\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.243865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.243838    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:15.641016 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:15.640981    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4bea89_c2b7_482f_a4d5_3c7b6b75de4e.slice/crio-c7f5a57cbcd9bc2c55ef0ed84fe54ebe9f197e71aa7a0aaafe3766343a14e2a3 WatchSource:0}: Error finding container c7f5a57cbcd9bc2c55ef0ed84fe54ebe9f197e71aa7a0aaafe3766343a14e2a3: Status 404 returned error can't find the container with id c7f5a57cbcd9bc2c55ef0ed84fe54ebe9f197e71aa7a0aaafe3766343a14e2a3
Oct 30 20:08:15.641840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.641815    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"]
Oct 30 20:08:15.950844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:15.950801    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf" event={"ID":"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e","Type":"ContainerStarted","Data":"c7f5a57cbcd9bc2c55ef0ed84fe54ebe9f197e71aa7a0aaafe3766343a14e2a3"}
Oct 30 20:08:16.960963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:16.960928    4917 generic.go:356] "Generic (PLEG): container finished" podID="db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" containerID="b41a35b656d444454f1323e81dee609eeddcbdec53564281d8f6150ba1f855f0" exitCode=0
Oct 30 20:08:16.960963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:16.960969    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf" event={"ID":"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e","Type":"ContainerDied","Data":"b41a35b656d444454f1323e81dee609eeddcbdec53564281d8f6150ba1f855f0"}
Oct 30 20:08:18.140396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.140370    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:18.303154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.303072    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca\") pod \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") "
Oct 30 20:08:18.303302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.303191    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca\") pod \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") "
Oct 30 20:08:18.303302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.303263    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg\" (UniqueName: \"kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg\") pod \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\" (UID: \"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e\") "
Oct 30 20:08:18.303512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.303483    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca" pod "db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" (UID: "db4bea89-c2b7-482f-a4d5-3c7b6b75de4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:18.303797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.303765    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:18.305299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.305277    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca" pod "db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" (UID: "db4bea89-c2b7-482f-a4d5-3c7b6b75de4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:18.305415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.305390    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg" pod "db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" (UID: "db4bea89-c2b7-482f-a4d5-3c7b6b75de4e"). InnerVolumeSpecName "kube-api-access-p2gqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:18.405648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.405617    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:18.405648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.405645    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gqg\" (UniqueName: \"kubernetes.io/projected/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e-kube-api-access-p2gqg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:18.978971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.978936    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"
Oct 30 20:08:18.979131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.978970    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf" event={"ID":"db4bea89-c2b7-482f-a4d5-3c7b6b75de4e","Type":"ContainerDied","Data":"c7f5a57cbcd9bc2c55ef0ed84fe54ebe9f197e71aa7a0aaafe3766343a14e2a3"}
Oct 30 20:08:18.979131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:18.979021    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f5a57cbcd9bc2c55ef0ed84fe54ebe9f197e71aa7a0aaafe3766343a14e2a3"
Oct 30 20:08:21.426362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:21.426321    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:21.439357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:21.439331    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:22.016008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:22.015981    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:22.710238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:22.710203    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"]
Oct 30 20:08:22.710665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:22.710482    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy" containerID="cri-o://5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa" gracePeriod=30
Oct 30 20:08:22.762938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:22.762902    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"]
Oct 30 20:08:22.763187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:22.763139    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy" containerID="cri-o://b1b4b578e2e1f45e6092cdb1badf249c279921785b1b3e4ac9b7041d2109e376" gracePeriod=30
Oct 30 20:08:28.065871 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.065835    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:08:28.067913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.067887    4917 generic.go:356] "Generic (PLEG): container finished" podID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerID="b1b4b578e2e1f45e6092cdb1badf249c279921785b1b3e4ac9b7041d2109e376" exitCode=0
Oct 30 20:08:28.068050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.067955    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" event={"ID":"c0df23aa-527b-4f03-b269-ff5acf36726d","Type":"ContainerDied","Data":"b1b4b578e2e1f45e6092cdb1badf249c279921785b1b3e4ac9b7041d2109e376"}
Oct 30 20:08:28.069660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.069628    4917 generic.go:356] "Generic (PLEG): container finished" podID="484556cc-6624-446f-a8c8-949ab75c31e2" containerID="5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa" exitCode=0
Oct 30 20:08:28.069749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.069675    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"
Oct 30 20:08:28.069749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.069709    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" event={"ID":"484556cc-6624-446f-a8c8-949ab75c31e2","Type":"ContainerDied","Data":"5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa"}
Oct 30 20:08:28.069869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.069755    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6" event={"ID":"484556cc-6624-446f-a8c8-949ab75c31e2","Type":"ContainerDied","Data":"28b87120d8fe5e6f460193ff5fc66ff18cdc815fc0e04f80eb39c5464ca7a436"}
Oct 30 20:08:28.069869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.069816    4917 scope.go:122] "RemoveContainer" containerID="5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa"
Oct 30 20:08:28.090872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.090852    4917 scope.go:122] "RemoveContainer" containerID="5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa"
Oct 30 20:08:28.091261 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:08:28.091124    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa\": container with ID starting with 5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa not found: ID does not exist" containerID="5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa"
Oct 30 20:08:28.091261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.091156    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa"} err="failed to get container status \"5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa\": rpc error: code = NotFound desc = could not find container \"5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa\": container with ID starting with 5d5fb717ece7ad5c553115d9de1baf42b54b03dcfdf0503a4f8910ac1c9abdaa not found: ID does not exist"
Oct 30 20:08:28.092455 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.092439    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:08:28.144884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.144859    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145037 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.144890    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145037 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.144938    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145037 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.144973    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.145206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145076    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.145206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145134    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145168    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145223    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.145358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145260    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.145358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145269    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.145358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145281    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:28.145358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145276    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.145358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145350    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145462    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.145648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145519    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.145648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145534    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.145648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145539    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.145867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145704    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145708    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.145867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145764    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.145867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145826    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.146071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145874    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.146071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145911    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo\") pod \"484556cc-6624-446f-a8c8-949ab75c31e2\" (UID: \"484556cc-6624-446f-a8c8-949ab75c31e2\") "
Oct 30 20:08:28.146071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145975    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.146071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.145980    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.146271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146122    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo\") pod \"c0df23aa-527b-4f03-b269-ff5acf36726d\" (UID: \"c0df23aa-527b-4f03-b269-ff5acf36726d\") "
Oct 30 20:08:28.146271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146229    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:28.146455 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146434    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146868    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146894    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146910    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146926    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146940    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0df23aa-527b-4f03-b269-ff5acf36726d-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146954    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484556cc-6624-446f-a8c8-949ab75c31e2-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146967    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146980    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.146993    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.147291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.147006    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.148178 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.148151    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:28.148278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.148212    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:28.148278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.148222    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "kube-api-access-wz97r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:28.148278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.148237    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.148862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.148831    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "kube-api-access-cdp48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:28.148862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.148849    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:08:28.149631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.149614    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo" pod "c0df23aa-527b-4f03-b269-ff5acf36726d" (UID: "c0df23aa-527b-4f03-b269-ff5acf36726d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:08:28.149995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.149978    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy" pod "484556cc-6624-446f-a8c8-949ab75c31e2" (UID: "484556cc-6624-446f-a8c8-949ab75c31e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248056    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248088    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484556cc-6624-446f-a8c8-949ab75c31e2-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248102    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz97r\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-kube-api-access-wz97r\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248120    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdp48\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-kube-api-access-cdp48\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248135    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484556cc-6624-446f-a8c8-949ab75c31e2-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248148    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248161    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484556cc-6624-446f-a8c8-949ab75c31e2-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.248171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.248177    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0df23aa-527b-4f03-b269-ff5acf36726d-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.592746 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.592671    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"]
Oct 30 20:08:28.630793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:28.614571    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8rll6"]
Oct 30 20:08:29.093798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:29.093324    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j" event={"ID":"c0df23aa-527b-4f03-b269-ff5acf36726d","Type":"ContainerDied","Data":"cf49d7b87fce95293d83a8ad2a1995476f58ec9a8f46ce212d407eb8f2755924"}
Oct 30 20:08:29.093798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:29.093418    4917 scope.go:122] "RemoveContainer" containerID="b1b4b578e2e1f45e6092cdb1badf249c279921785b1b3e4ac9b7041d2109e376"
Oct 30 20:08:29.093798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:29.093573    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"
Oct 30 20:08:29.289792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:29.289592    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"]
Oct 30 20:08:29.305795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:29.305412    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pqs8j"]
Oct 30 20:08:30.183797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.182242    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" path="/var/lib/kubelet/pods/484556cc-6624-446f-a8c8-949ab75c31e2/volumes"
Oct 30 20:08:30.196793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.195271    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" path="/var/lib/kubelet/pods/c0df23aa-527b-4f03-b269-ff5acf36726d/volumes"
Oct 30 20:08:30.531844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.530909    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.653794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.652878    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.678797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.673922    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.713796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.712866    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 30 20:08:30.734795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.718373    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.774797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.765887    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.774797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.766407    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.774797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.767335    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.774797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.767671    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.774797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.774255    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.808860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.798870    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.821803 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:30.821494    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.098963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.098857    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.124925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.124866    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.168083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.167926    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.253922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.253869    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.549795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.546394    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.558793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.554228    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:08:31.653146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.653101    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.680795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.676972    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:08:31.703818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.693869    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.703818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.694332    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:08:31.718919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.718870    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.745023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.744985    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:08:31.773901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.773852    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:08:31.794795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.774446    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:08:31.820188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.777643    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:08:31.827795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.820459    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:08:31.854863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.778760    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.855829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.778798    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.856054    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.856124    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.803870    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.778728    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.821741    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.858477    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.858524    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:08:31.884810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:31.858559    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:08:32.030805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.027226    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.110287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.110188    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.158796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.128307    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.198831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.173873    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.384853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.326170    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:08:32.449818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.445938    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.474796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.450187    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:32.475229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.475199    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"]
Oct 30 20:08:32.501817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.480699    4917 cpu_manager.go:414] "containerMap: removing container" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy"
Oct 30 20:08:32.543870 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.543834    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy"
Oct 30 20:08:32.595025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.594993    4917 cpu_manager.go:414] "containerMap: removing container" podUID="db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" containerName="revoke-keys"
Oct 30 20:08:32.595229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.595212    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" containerName="revoke-keys"
Oct 30 20:08:32.595372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.595358    4917 cpu_manager.go:414] "containerMap: removing container" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy"
Oct 30 20:08:32.595465 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.595452    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy"
Oct 30 20:08:32.620795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.595827    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0df23aa-527b-4f03-b269-ff5acf36726d" containerName="istio-proxy"
Oct 30 20:08:32.622272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.622239    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" containerName="revoke-keys"
Oct 30 20:08:32.641909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.641821    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="484556cc-6624-446f-a8c8-949ab75c31e2" containerName="istio-proxy"
Oct 30 20:08:32.644840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.555881    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.675552 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.675507    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.715794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.712009    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.750794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.724249    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.786855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.786811    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.829796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.822721    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.829796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.825216    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 20:08:32.829796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.826604    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:08:32.829796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.827128    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:32.874498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.856026    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"]
Oct 30 20:08:32.874498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.856585    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.874498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.871655    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.897667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.897567    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.900079 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.897506    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.940825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:32.940764    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.145792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.135576    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.236152 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.141848    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.244977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.189825    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.235943    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258462    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258542    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826xx\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258575    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258685    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258720    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258861    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.258990    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.262807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.259063    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.402734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.402692    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.403358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.403332    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.403616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.403595    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.403879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.403857    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.404049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.404030    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.404228 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.404212    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.404423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.404402    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.404591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.404575    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826xx\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.404705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.404686    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.492681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.454348    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.492681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.455070    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.492681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.490198    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.492681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.492135    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.493485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.493461    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.557805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.534378    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.558062 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.547277    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.578739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.578703    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-826xx\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.646268 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.646225    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.683430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.683379    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.712572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.712537    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.730851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.728267    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.754880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.752866    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.811994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.805099    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.826914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.815998    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:33.855684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.855626    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.941813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.880588    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.891219    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.914856    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.916078    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:33.942642    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.024290    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.024360    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.257014    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.266060    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.359807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.290314    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.710793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.650074    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.710793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.686651    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.728902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.728854    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.814886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.769024    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.815614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.815573    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.876978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:34.876930    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.417793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.417196    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.473795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.470150    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.482501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.482459    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.503055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.483256    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.503055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.483365    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.503055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.483454    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.503055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.483558    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.503055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.483668    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.539000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.538950    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:08:35.641356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.641327    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:08:35.651222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.651160    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.686910 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.686861    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.853503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.853455    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.858900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.858865    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.872724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:35.872687    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.006182 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:35.997249    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608c759b_1b44_444e_a4b1_6b1482b59f59.slice/crio-56cd7f9beca70a5a002a8415ebbffa338ac38ef4120b82b8421ee1767f212c5f WatchSource:0}: Error finding container 56cd7f9beca70a5a002a8415ebbffa338ac38ef4120b82b8421ee1767f212c5f: Status 404 returned error can't find the container with id 56cd7f9beca70a5a002a8415ebbffa338ac38ef4120b82b8421ee1767f212c5f
Oct 30 20:08:36.021740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.020935    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:36.021740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.021056    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:36.021740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.021106    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:36.079859 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.079827    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"]
Oct 30 20:08:36.086302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.086280    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:08:36.103921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.103895    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:08:36.196006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.159385    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:08:36.208807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.207060    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:08:36.208807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.207903    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:08:36.208807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.208621    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:08:36.255391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.253539    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:08:36.266976 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.263253    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:08:36.267958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.267937    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:08:36.268483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.268464    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:08:36.273945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.273644    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:08:36.308905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.308878    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:08:36.660498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.660408    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" event={"ID":"608c759b-1b44-444e-a4b1-6b1482b59f59","Type":"ContainerStarted","Data":"542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708"}
Oct 30 20:08:36.660498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.660466    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" event={"ID":"608c759b-1b44-444e-a4b1-6b1482b59f59","Type":"ContainerStarted","Data":"56cd7f9beca70a5a002a8415ebbffa338ac38ef4120b82b8421ee1767f212c5f"}
Oct 30 20:08:36.827627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:36.827598    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:37.828866 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:37.828833    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 20:08:37.829344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:37.828891    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 20:08:38.827810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:38.827764    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 20:08:38.828070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:38.827835    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 20:08:39.691518 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:08:39.691477    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:08:39.828231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:39.828193    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 20:08:39.828395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:39.828267    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 20:08:40.827609 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:40.827569    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 20:08:40.828108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:40.827642    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 20:08:41.828113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:41.828076    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 20:08:41.828554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:41.828179    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 20:08:42.667696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.667643    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podStartSLOduration=10.667627438 podStartE2EDuration="10.667627438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:37.716252092 +0000 UTC m=+1613.880653734" watchObservedRunningTime="2026-10-30 20:08:42.667627438 +0000 UTC m=+1618.832029097"
Oct 30 20:08:42.670391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.670365    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"]
Oct 30 20:08:42.675122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.675102    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.709699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.709666    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.709912 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.709890    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.710074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.710055    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpv2\" (UniqueName: \"kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.735240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.735206    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"]
Oct 30 20:08:42.779567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.779536    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:42.779735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.779569    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:08:42.779735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.779577    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:08:42.783019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.783001    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:42.811311 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.811284    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpv2\" (UniqueName: \"kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.811456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.811398    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.811570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.811554    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.812240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.812217    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.814070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.814048    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.828493 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.828471    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 20:08:42.828884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.828531    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 20:08:42.879109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.879076    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpv2\" (UniqueName: \"kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2\") pod \"maas-api-revoke-sub-cd16f2ae-7gdrp\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:42.990169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:42.990137    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:43.373036 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:43.372994    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3893bc5_d086_41bf_9139_2f541f66e76a.slice/crio-db9ac9f1a1fa865306431c61145c7548021b9d6eb25ca4e7408ebb95cee3980f WatchSource:0}: Error finding container db9ac9f1a1fa865306431c61145c7548021b9d6eb25ca4e7408ebb95cee3980f: Status 404 returned error can't find the container with id db9ac9f1a1fa865306431c61145c7548021b9d6eb25ca4e7408ebb95cee3980f
Oct 30 20:08:43.373036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:43.373026    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"]
Oct 30 20:08:43.739658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:43.739616    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp" event={"ID":"b3893bc5-d086-41bf-9139-2f541f66e76a","Type":"ContainerStarted","Data":"b1206e40451bf2a8745e1e32ea6248488336c012e9c6207be1dd3df2ea73b9b6"}
Oct 30 20:08:43.739886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:43.739704    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp" event={"ID":"b3893bc5-d086-41bf-9139-2f541f66e76a","Type":"ContainerStarted","Data":"db9ac9f1a1fa865306431c61145c7548021b9d6eb25ca4e7408ebb95cee3980f"}
Oct 30 20:08:43.857940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:43.857888    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp" podStartSLOduration=1.8578708750000001 podStartE2EDuration="1.857870875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:43.856986649 +0000 UTC m=+1620.021388292" watchObservedRunningTime="2026-10-30 20:08:43.857870875 +0000 UTC m=+1620.022272550"
Oct 30 20:08:44.047484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.047453    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:44.048047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.048030    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:44.049142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.049117    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:08:44.275352 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.275319    4917 scope.go:122] "RemoveContainer" containerID="f6fe0265e3a6bf3024c5e6738fda2b4ef1b8626f232bfcf71fe60566b37c5419"
Oct 30 20:08:44.285716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.285696    4917 scope.go:122] "RemoveContainer" containerID="00c648f7b2789f327c0ecd457f571683989c110094e926fcd2ab8a0e71058a4f"
Oct 30 20:08:44.295787 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.295726    4917 scope.go:122] "RemoveContainer" containerID="2221b186a5245fa32bb38d29c65812d8c87fbf75e37960036d7e9324cde90567"
Oct 30 20:08:44.306685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.306659    4917 scope.go:122] "RemoveContainer" containerID="9fa3e236e9676818e2f003e699f5f815ac3b5849d1ec7897aafe6355e878c7b6"
Oct 30 20:08:44.308272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.308239    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"]
Oct 30 20:08:44.314589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.314565    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.319603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.319583    4917 scope.go:122] "RemoveContainer" containerID="1c16479dbec9205c5f289d592f77da682d552d3a374907565094ba9ca32d21d2"
Oct 30 20:08:44.383222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.383195    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"]
Oct 30 20:08:44.389934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.389915    4917 scope.go:122] "RemoveContainer" containerID="0c78bc7df57e041bbf414a874c3922f701c24818546112a266c7a2ed752d7e4d"
Oct 30 20:08:44.403218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.403191    4917 scope.go:122] "RemoveContainer" containerID="76e42818c8ed3464d0fbdb11c57f9f551ed8bc786866bd26d599001f51155c2f"
Oct 30 20:08:44.436319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.436283    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4d5v\" (UniqueName: \"kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.436485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.436435    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.436560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.436537    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.538348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.538317    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4d5v\" (UniqueName: \"kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.538487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.538396    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.538487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.538450    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.539064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.539043    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.540914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.540891    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.600949 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.600893    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4d5v\" (UniqueName: \"kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v\") pod \"maas-api-revoke-sub-c2b8bc59-pv2hh\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.687579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.687543    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:44.757556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.754540    4917 generic.go:356] "Generic (PLEG): container finished" podID="b3893bc5-d086-41bf-9139-2f541f66e76a" containerID="b1206e40451bf2a8745e1e32ea6248488336c012e9c6207be1dd3df2ea73b9b6" exitCode=0
Oct 30 20:08:44.757556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:44.755977    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp" event={"ID":"b3893bc5-d086-41bf-9139-2f541f66e76a","Type":"ContainerDied","Data":"b1206e40451bf2a8745e1e32ea6248488336c012e9c6207be1dd3df2ea73b9b6"}
Oct 30 20:08:45.070252 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:45.070223    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2204ec6_3fec_44da_9db9_02f1e06d0a83.slice/crio-dfe3c274b587fb51cd8c5524597e8dae40a27ae3b476b269972a1f19ebd55d49 WatchSource:0}: Error finding container dfe3c274b587fb51cd8c5524597e8dae40a27ae3b476b269972a1f19ebd55d49: Status 404 returned error can't find the container with id dfe3c274b587fb51cd8c5524597e8dae40a27ae3b476b269972a1f19ebd55d49
Oct 30 20:08:45.071848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:45.071824    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"]
Oct 30 20:08:45.765843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:45.765806    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh" event={"ID":"c2204ec6-3fec-44da-9db9-02f1e06d0a83","Type":"ContainerStarted","Data":"351f640aeeb424d9495891dd0060bc97c69d5f3cc6b6c2605125e17163ad116b"}
Oct 30 20:08:45.766040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:45.765867    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh" event={"ID":"c2204ec6-3fec-44da-9db9-02f1e06d0a83","Type":"ContainerStarted","Data":"dfe3c274b587fb51cd8c5524597e8dae40a27ae3b476b269972a1f19ebd55d49"}
Oct 30 20:08:45.885287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:45.885207    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh" podStartSLOduration=1.8851878210000002 podStartE2EDuration="1.885187821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:45.884597567 +0000 UTC m=+1622.048999211" watchObservedRunningTime="2026-10-30 20:08:45.885187821 +0000 UTC m=+1622.049589466"
Oct 30 20:08:45.944562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:45.944536    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:46.059521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.059486    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca\") pod \"b3893bc5-d086-41bf-9139-2f541f66e76a\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") "
Oct 30 20:08:46.059672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.059629    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca\") pod \"b3893bc5-d086-41bf-9139-2f541f66e76a\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") "
Oct 30 20:08:46.059798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.059753    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2\" (UniqueName: \"kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2\") pod \"b3893bc5-d086-41bf-9139-2f541f66e76a\" (UID: \"b3893bc5-d086-41bf-9139-2f541f66e76a\") "
Oct 30 20:08:46.059993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.059969    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca" pod "b3893bc5-d086-41bf-9139-2f541f66e76a" (UID: "b3893bc5-d086-41bf-9139-2f541f66e76a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:46.060275 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.060251    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:46.061514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.061494    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca" pod "b3893bc5-d086-41bf-9139-2f541f66e76a" (UID: "b3893bc5-d086-41bf-9139-2f541f66e76a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:46.061742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.061720    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2" pod "b3893bc5-d086-41bf-9139-2f541f66e76a" (UID: "b3893bc5-d086-41bf-9139-2f541f66e76a"). InnerVolumeSpecName "kube-api-access-lxpv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:46.161731 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.161701    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3893bc5-d086-41bf-9139-2f541f66e76a-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:46.161731 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.161729    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxpv2\" (UniqueName: \"kubernetes.io/projected/b3893bc5-d086-41bf-9139-2f541f66e76a-kube-api-access-lxpv2\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:46.776296 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.776250    4917 generic.go:356] "Generic (PLEG): container finished" podID="c2204ec6-3fec-44da-9db9-02f1e06d0a83" containerID="351f640aeeb424d9495891dd0060bc97c69d5f3cc6b6c2605125e17163ad116b" exitCode=0
Oct 30 20:08:46.776503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.776340    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh" event={"ID":"c2204ec6-3fec-44da-9db9-02f1e06d0a83","Type":"ContainerDied","Data":"351f640aeeb424d9495891dd0060bc97c69d5f3cc6b6c2605125e17163ad116b"}
Oct 30 20:08:46.777824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.777798    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp" event={"ID":"b3893bc5-d086-41bf-9139-2f541f66e76a","Type":"ContainerDied","Data":"db9ac9f1a1fa865306431c61145c7548021b9d6eb25ca4e7408ebb95cee3980f"}
Oct 30 20:08:46.777824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.777825    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"
Oct 30 20:08:46.777986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:46.777840    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db9ac9f1a1fa865306431c61145c7548021b9d6eb25ca4e7408ebb95cee3980f"
Oct 30 20:08:47.970892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:47.970868    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:48.084672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.084599    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"]
Oct 30 20:08:48.085303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.085283    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b3893bc5-d086-41bf-9139-2f541f66e76a" containerName="revoke-keys"
Oct 30 20:08:48.085415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.085303    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3893bc5-d086-41bf-9139-2f541f66e76a" containerName="revoke-keys"
Oct 30 20:08:48.085415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.085326    4917 cpu_manager.go:414] "containerMap: removing container" podUID="c2204ec6-3fec-44da-9db9-02f1e06d0a83" containerName="revoke-keys"
Oct 30 20:08:48.085415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.085335    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2204ec6-3fec-44da-9db9-02f1e06d0a83" containerName="revoke-keys"
Oct 30 20:08:48.085575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.085513    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2204ec6-3fec-44da-9db9-02f1e06d0a83" containerName="revoke-keys"
Oct 30 20:08:48.085575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.085540    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3893bc5-d086-41bf-9139-2f541f66e76a" containerName="revoke-keys"
Oct 30 20:08:48.087028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.087003    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca\") pod \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") "
Oct 30 20:08:48.087181 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.087143    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca\") pod \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") "
Oct 30 20:08:48.087248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.087210    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v\" (UniqueName: \"kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v\") pod \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\" (UID: \"c2204ec6-3fec-44da-9db9-02f1e06d0a83\") "
Oct 30 20:08:48.087460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.087432    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca" pod "c2204ec6-3fec-44da-9db9-02f1e06d0a83" (UID: "c2204ec6-3fec-44da-9db9-02f1e06d0a83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:48.088374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.088345    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:48.089750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.089719    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v" pod "c2204ec6-3fec-44da-9db9-02f1e06d0a83" (UID: "c2204ec6-3fec-44da-9db9-02f1e06d0a83"). InnerVolumeSpecName "kube-api-access-x4d5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:48.089999 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.089937    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.096074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.096044    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca" pod "c2204ec6-3fec-44da-9db9-02f1e06d0a83" (UID: "c2204ec6-3fec-44da-9db9-02f1e06d0a83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:48.112595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.112565    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"]
Oct 30 20:08:48.189922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.189888    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.189922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.189926    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.189944    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190126    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190251    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190314    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190395    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4tj\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190460    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190481    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.190603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190592    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2204ec6-3fec-44da-9db9-02f1e06d0a83-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:48.190641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.190606    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4d5v\" (UniqueName: \"kubernetes.io/projected/c2204ec6-3fec-44da-9db9-02f1e06d0a83-kube-api-access-x4d5v\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:48.291918 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.291880    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4tj\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.291966    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.291990    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292052    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292096    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292115    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292198    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292291    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292346    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292446    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292504    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292556    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292702    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.292861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.292801    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.294634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.294607    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.294865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.294845    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.355134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.355053    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4tj\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.355473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.355455    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.456340 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.456305    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:48.800374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.800338    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"
Oct 30 20:08:48.800554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.800344    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh" event={"ID":"c2204ec6-3fec-44da-9db9-02f1e06d0a83","Type":"ContainerDied","Data":"dfe3c274b587fb51cd8c5524597e8dae40a27ae3b476b269972a1f19ebd55d49"}
Oct 30 20:08:48.800554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.800492    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe3c274b587fb51cd8c5524597e8dae40a27ae3b476b269972a1f19ebd55d49"
Oct 30 20:08:48.904411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.904379    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"]
Oct 30 20:08:48.924024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.923993    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:48.924118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.924060    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:48.924118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:48.924091    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:49.002810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.002781    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"]
Oct 30 20:08:49.008460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.008439    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.071172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.071086    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"]
Oct 30 20:08:49.121705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.121665    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcx2v\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.121879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.121762    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.121934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.121894    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.121975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.121936    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.122015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.121988    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.122101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.122085    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.122135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.122125    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.122166 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.122154    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.122201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.122180    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.223634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.223590    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.223634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.223636    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.223955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.223667    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.223955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.223757    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcx2v\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.223955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.223852    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.223965    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224009    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224064    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224131    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224131    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224210    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224228    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224431 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224417    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.224737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.224714    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.226542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.226516    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.226723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.226706    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.295384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.295349    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.296076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.295567    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcx2v\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.324131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.324051    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:49.772845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.772816    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"]
Oct 30 20:08:49.775156 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:49.775129    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52b2056_acf0_4ab9_be03_f354baf75ba3.slice/crio-2cae848d9966bd9594ca42ba5fe681e95126f3f0c3513d04a5670afb8045b4c2 WatchSource:0}: Error finding container 2cae848d9966bd9594ca42ba5fe681e95126f3f0c3513d04a5670afb8045b4c2: Status 404 returned error can't find the container with id 2cae848d9966bd9594ca42ba5fe681e95126f3f0c3513d04a5670afb8045b4c2
Oct 30 20:08:49.777803 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.777736    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:49.777895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.777836    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:49.777895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.777867    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:08:49.811976 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.811950    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" event={"ID":"39b2dce0-514d-4ab3-8b24-5106f5613b97","Type":"ContainerStarted","Data":"dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc"}
Oct 30 20:08:49.812176 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.812148    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" event={"ID":"39b2dce0-514d-4ab3-8b24-5106f5613b97","Type":"ContainerStarted","Data":"d059c34b12078b83906f7cd1dee4e823449b32ebbd468ed86350c9f7bd0619e2"}
Oct 30 20:08:49.813732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.813711    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" event={"ID":"d52b2056-acf0-4ab9-be03-f354baf75ba3","Type":"ContainerStarted","Data":"2cae848d9966bd9594ca42ba5fe681e95126f3f0c3513d04a5670afb8045b4c2"}
Oct 30 20:08:49.966855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:49.966791    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podStartSLOduration=2.966748013 podStartE2EDuration="2.966748013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:49.938108658 +0000 UTC m=+1626.102510301" watchObservedRunningTime="2026-10-30 20:08:49.966748013 +0000 UTC m=+1626.131149656"
Oct 30 20:08:50.467794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:50.461269    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:50.478911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:50.478874    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:08:50.479146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:50.479122    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:08:50.823822 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:50.823717    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" event={"ID":"d52b2056-acf0-4ab9-be03-f354baf75ba3","Type":"ContainerStarted","Data":"4dc3873877aa16f36463dcc68e3d136c018251cc4adc2c901fd1964cb7e7afb7"}
Oct 30 20:08:51.092879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:51.092732    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podStartSLOduration=3.09270969 podStartE2EDuration="3.09270969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:50.996230805 +0000 UTC m=+1627.160632470" watchObservedRunningTime="2026-10-30 20:08:51.09270969 +0000 UTC m=+1627.257111330"
Oct 30 20:08:51.324339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:51.324307    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:51.325788 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:51.325748    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:08:51.325913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:51.325813    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:08:51.457763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:51.457725    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:08:51.458255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:51.458122    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:08:52.325390 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:52.325355    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:08:52.325843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:52.325410    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:08:52.467317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:52.467278    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:08:52.467599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:52.467566    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:08:53.324843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.324813    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:08:53.325084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.325059    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:08:53.459421 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.459386    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:08:53.460036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.460004    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:08:53.493819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.491663    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"]
Oct 30 20:08:53.497381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.497357    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.529569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.529541    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7lt\" (UniqueName: \"kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.529716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.529697    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.529974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.529823    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.539914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.539881    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"]
Oct 30 20:08:53.571827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.568230    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:53.571827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.568318    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:53.571827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.569034    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:08:53.593853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.584961    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:08:53.631239 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.631098    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.631561 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.631301    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7lt\" (UniqueName: \"kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.631561 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.631427    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.632798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.632447    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.633597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.633576    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.698093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.697957    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7lt\" (UniqueName: \"kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt\") pod \"maas-api-revoke-sub-bef8074a-59lcx\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:53.814004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:53.813969    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:54.301033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.301001    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"]
Oct 30 20:08:54.324978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.324949    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:08:54.325061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.325005    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:08:54.458379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.458151    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:08:54.458379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.458213    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:08:54.480794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.480476    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"]
Oct 30 20:08:54.494792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.494020    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.533545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.533513    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"]
Oct 30 20:08:54.608372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.602893    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjc4\" (UniqueName: \"kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.608372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.602957    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.608372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.603018    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.708234 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.707333    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjc4\" (UniqueName: \"kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.708234 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.707404    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.708234 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.707479    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.709382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.709350    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.714153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.714131    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.783794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.775445    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjc4\" (UniqueName: \"kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4\") pod \"maas-api-revoke-sub-083d6d38-t45mx\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.818692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.818658    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:54.925696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:54.924445    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx" event={"ID":"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52","Type":"ContainerStarted","Data":"98541a36ab8926f3776fa451e1106be9aa974ba8c3fdb86cdf4bc844aac694dc"}
Oct 30 20:08:55.288503 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:08:55.288472    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1455b048_4f80_431a_ae23_8ecd28bf9478.slice/crio-ae7fc17164eef94b792852b9ba422a51dbfa4c4faba814abe6ee6a10f94b4144 WatchSource:0}: Error finding container ae7fc17164eef94b792852b9ba422a51dbfa4c4faba814abe6ee6a10f94b4144: Status 404 returned error can't find the container with id ae7fc17164eef94b792852b9ba422a51dbfa4c4faba814abe6ee6a10f94b4144
Oct 30 20:08:55.290972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.290945    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"]
Oct 30 20:08:55.325606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.325579    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:08:55.325747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.325628    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:08:55.460434 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.460393    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:08:55.460688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.460656    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:08:55.935516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.935478    4917 generic.go:356] "Generic (PLEG): container finished" podID="7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" containerID="ec96acd74a746e26a3baee9a09779df948148f880e17540546861d0961a81293" exitCode=0
Oct 30 20:08:55.935951 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.935567    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx" event={"ID":"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52","Type":"ContainerDied","Data":"ec96acd74a746e26a3baee9a09779df948148f880e17540546861d0961a81293"}
Oct 30 20:08:55.937513 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.937490    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx" event={"ID":"1455b048-4f80-431a-ae23-8ecd28bf9478","Type":"ContainerStarted","Data":"5903c1a5baeaa7af89e6b58db4ee803b7eca7f9ee8fc176046ba8fa774fae6b5"}
Oct 30 20:08:55.937630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:55.937536    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx" event={"ID":"1455b048-4f80-431a-ae23-8ecd28bf9478","Type":"ContainerStarted","Data":"ae7fc17164eef94b792852b9ba422a51dbfa4c4faba814abe6ee6a10f94b4144"}
Oct 30 20:08:56.173716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.173648    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx" podStartSLOduration=2.173631255 podStartE2EDuration="2.173631255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:56.172108493 +0000 UTC m=+1632.336510135" watchObservedRunningTime="2026-10-30 20:08:56.173631255 +0000 UTC m=+1632.338032897"
Oct 30 20:08:56.324675 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.324584    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:08:56.324675 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.324650    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:08:56.626678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.626586    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:56.627563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.627499    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:56.628293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.628270    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:08:56.950118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.950075    4917 generic.go:356] "Generic (PLEG): container finished" podID="1455b048-4f80-431a-ae23-8ecd28bf9478" containerID="5903c1a5baeaa7af89e6b58db4ee803b7eca7f9ee8fc176046ba8fa774fae6b5" exitCode=0
Oct 30 20:08:56.950615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:56.950212    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx" event={"ID":"1455b048-4f80-431a-ae23-8ecd28bf9478","Type":"ContainerDied","Data":"5903c1a5baeaa7af89e6b58db4ee803b7eca7f9ee8fc176046ba8fa774fae6b5"}
Oct 30 20:08:57.161198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.161172    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:57.253088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.252994    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt\" (UniqueName: \"kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt\") pod \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") "
Oct 30 20:08:57.253256 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.253187    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca\") pod \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") "
Oct 30 20:08:57.253407 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.253379    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca\") pod \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\" (UID: \"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52\") "
Oct 30 20:08:57.254404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.254376    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca" pod "7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" (UID: "7f964f34-2bdb-46ce-aa0c-1ca959d9fc52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:57.257611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.257581    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt" pod "7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" (UID: "7f964f34-2bdb-46ce-aa0c-1ca959d9fc52"). InnerVolumeSpecName "kube-api-access-xf7lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:57.270597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.261304    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca" pod "7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" (UID: "7f964f34-2bdb-46ce-aa0c-1ca959d9fc52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:57.328715 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.328681    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:57.329233 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.329208    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:57.329907 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.329887    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:08:57.354941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.354910    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:57.354941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.354940    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:57.354941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.354956    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7lt\" (UniqueName: \"kubernetes.io/projected/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52-kube-api-access-xf7lt\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:57.960295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.960261    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"
Oct 30 20:08:57.960801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.960285    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx" event={"ID":"7f964f34-2bdb-46ce-aa0c-1ca959d9fc52","Type":"ContainerDied","Data":"98541a36ab8926f3776fa451e1106be9aa974ba8c3fdb86cdf4bc844aac694dc"}
Oct 30 20:08:57.960801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:57.960360    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98541a36ab8926f3776fa451e1106be9aa974ba8c3fdb86cdf4bc844aac694dc"
Oct 30 20:08:58.145623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.145597    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:58.268839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.268166    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4\" (UniqueName: \"kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4\") pod \"1455b048-4f80-431a-ae23-8ecd28bf9478\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") "
Oct 30 20:08:58.268839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.268276    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca\") pod \"1455b048-4f80-431a-ae23-8ecd28bf9478\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") "
Oct 30 20:08:58.268839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.268391    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca\") pod \"1455b048-4f80-431a-ae23-8ecd28bf9478\" (UID: \"1455b048-4f80-431a-ae23-8ecd28bf9478\") "
Oct 30 20:08:58.276958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.270197    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca" pod "1455b048-4f80-431a-ae23-8ecd28bf9478" (UID: "1455b048-4f80-431a-ae23-8ecd28bf9478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:58.276958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.274693    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4" pod "1455b048-4f80-431a-ae23-8ecd28bf9478" (UID: "1455b048-4f80-431a-ae23-8ecd28bf9478"). InnerVolumeSpecName "kube-api-access-lhjc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:58.276958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.274859    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca" pod "1455b048-4f80-431a-ae23-8ecd28bf9478" (UID: "1455b048-4f80-431a-ae23-8ecd28bf9478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:58.369817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.369787    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjc4\" (UniqueName: \"kubernetes.io/projected/1455b048-4f80-431a-ae23-8ecd28bf9478-kube-api-access-lhjc4\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:58.369817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.369815    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:58.369985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.369826    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1455b048-4f80-431a-ae23-8ecd28bf9478-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:58.970623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.970596    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"
Oct 30 20:08:58.971043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.970617    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx" event={"ID":"1455b048-4f80-431a-ae23-8ecd28bf9478","Type":"ContainerDied","Data":"ae7fc17164eef94b792852b9ba422a51dbfa4c4faba814abe6ee6a10f94b4144"}
Oct 30 20:08:58.971043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:08:58.970668    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7fc17164eef94b792852b9ba422a51dbfa4c4faba814abe6ee6a10f94b4144"
Oct 30 20:09:03.115826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.115792    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"]
Oct 30 20:09:03.116436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.116421    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" containerName="revoke-keys"
Oct 30 20:09:03.116486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.116437    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" containerName="revoke-keys"
Oct 30 20:09:03.116486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.116452    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1455b048-4f80-431a-ae23-8ecd28bf9478" containerName="revoke-keys"
Oct 30 20:09:03.116486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.116459    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1455b048-4f80-431a-ae23-8ecd28bf9478" containerName="revoke-keys"
Oct 30 20:09:03.116586 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.116566    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" containerName="revoke-keys"
Oct 30 20:09:03.116586 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.116575    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1455b048-4f80-431a-ae23-8ecd28bf9478" containerName="revoke-keys"
Oct 30 20:09:03.120485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.120467    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.220329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.220297    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"]
Oct 30 20:09:03.235291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235259    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235318    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235429    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235500    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235589    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235626    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsd7z\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235718    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235734    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.235789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.235764    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.337524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337485    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.337524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337524    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.337799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337554    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.337799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337627    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.337799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337680    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.337799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337755    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337860    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337946    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.337999    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsd7z\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.338036    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.338111    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.338188    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.338353    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.338650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.338624    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.340444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.340420    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.340663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.340644    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.399765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.399686    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.399932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.399859    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsd7z\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.435861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.435836    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:03.875923 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:09:03.875892    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b11eea_3dec_4bba_9b7c_140884a72905.slice/crio-6b4506024e7a1c58070e4fa21dca0784a156ef529f2c939a8400ce3759ac817a WatchSource:0}: Error finding container 6b4506024e7a1c58070e4fa21dca0784a156ef529f2c939a8400ce3759ac817a: Status 404 returned error can't find the container with id 6b4506024e7a1c58070e4fa21dca0784a156ef529f2c939a8400ce3759ac817a
Oct 30 20:09:03.877011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.876978    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"]
Oct 30 20:09:03.878643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.878608    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:09:03.878733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.878675    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:09:03.878733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:03.878705    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:09:04.014577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.014545    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" event={"ID":"24b11eea-3dec-4bba-9b7c-140884a72905","Type":"ContainerStarted","Data":"3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce"}
Oct 30 20:09:04.014790 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.014596    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" event={"ID":"24b11eea-3dec-4bba-9b7c-140884a72905","Type":"ContainerStarted","Data":"6b4506024e7a1c58070e4fa21dca0784a156ef529f2c939a8400ce3759ac817a"}
Oct 30 20:09:04.145110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.144986    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podStartSLOduration=1.144966506 podStartE2EDuration="1.144966506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:04.138865635 +0000 UTC m=+1640.303267278" watchObservedRunningTime="2026-10-30 20:09:04.144966506 +0000 UTC m=+1640.309368229"
Oct 30 20:09:04.427098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.427074    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"]
Oct 30 20:09:04.432282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.432256    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.436021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.435996    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:04.437800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.437765    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:04.437904 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.437820    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:04.485755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.485732    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:09:04.486092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.485812    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:09:04.486092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.485732    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:04.486092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.485950    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:09:04.489918 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.489894    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"]
Oct 30 20:09:04.556862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.556830    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.557018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.556868    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbqn\" (UniqueName: \"kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.557018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.556954    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.658706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.658667    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.658902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.658712    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbqn\" (UniqueName: \"kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.658902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.658759    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.659517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.659496    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.661101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.661080    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.721719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.721639    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbqn\" (UniqueName: \"kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn\") pod \"maas-api-revoke-sub-b298a625-4tbjf\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:04.769795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:04.765010    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:05.170624 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:09:05.170590    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5354a6_751b_4959_9982_f061b80cf271.slice/crio-feff21ea8e9e53c92023800b757f71f58216588da59c31d2975a98a6cb3fa31b WatchSource:0}: Error finding container feff21ea8e9e53c92023800b757f71f58216588da59c31d2975a98a6cb3fa31b: Status 404 returned error can't find the container with id feff21ea8e9e53c92023800b757f71f58216588da59c31d2975a98a6cb3fa31b
Oct 30 20:09:05.173235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:05.173206    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"]
Oct 30 20:09:05.436669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:05.436635    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:05.436856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:05.436689    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:06.032380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:06.032345    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf" event={"ID":"0d5354a6-751b-4959-9982-f061b80cf271","Type":"ContainerStarted","Data":"2337d092c37842f0ae99733c6e8469d070504ba0e7e530446fbf5d91e2225c9d"}
Oct 30 20:09:06.032545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:06.032399    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf" event={"ID":"0d5354a6-751b-4959-9982-f061b80cf271","Type":"ContainerStarted","Data":"feff21ea8e9e53c92023800b757f71f58216588da59c31d2975a98a6cb3fa31b"}
Oct 30 20:09:06.437057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:06.437005    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:06.437530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:06.437069    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:07.067920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:07.067882    4917 generic.go:356] "Generic (PLEG): container finished" podID="0d5354a6-751b-4959-9982-f061b80cf271" containerID="2337d092c37842f0ae99733c6e8469d070504ba0e7e530446fbf5d91e2225c9d" exitCode=0
Oct 30 20:09:07.068110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:07.067995    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf" event={"ID":"0d5354a6-751b-4959-9982-f061b80cf271","Type":"ContainerDied","Data":"2337d092c37842f0ae99733c6e8469d070504ba0e7e530446fbf5d91e2225c9d"}
Oct 30 20:09:07.169973 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:07.169903    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf" podStartSLOduration=3.169883889 podStartE2EDuration="3.169883889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:06.161290335 +0000 UTC m=+1642.325691978" watchObservedRunningTime="2026-10-30 20:09:07.169883889 +0000 UTC m=+1643.334285530"
Oct 30 20:09:07.177216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:07.177180    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:09:07.437110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:07.437080    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:07.437527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:07.437132    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:08.457620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.457584    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:08.484190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.484150    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:08.628794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.628146    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:08.773454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.773363    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca\") pod \"0d5354a6-751b-4959-9982-f061b80cf271\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") "
Oct 30 20:09:08.777992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.777956    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn\" (UniqueName: \"kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn\") pod \"0d5354a6-751b-4959-9982-f061b80cf271\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") "
Oct 30 20:09:08.778848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.778822    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca\") pod \"0d5354a6-751b-4959-9982-f061b80cf271\" (UID: \"0d5354a6-751b-4959-9982-f061b80cf271\") "
Oct 30 20:09:08.780206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.780181    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca" pod "0d5354a6-751b-4959-9982-f061b80cf271" (UID: "0d5354a6-751b-4959-9982-f061b80cf271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:08.795842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.794395    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca" pod "0d5354a6-751b-4959-9982-f061b80cf271" (UID: "0d5354a6-751b-4959-9982-f061b80cf271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:08.796496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.796451    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn" pod "0d5354a6-751b-4959-9982-f061b80cf271" (UID: "0d5354a6-751b-4959-9982-f061b80cf271"). InnerVolumeSpecName "kube-api-access-hfbqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:08.894082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.894036    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:08.894082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.894085    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfbqn\" (UniqueName: \"kubernetes.io/projected/0d5354a6-751b-4959-9982-f061b80cf271-kube-api-access-hfbqn\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:08.894343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:08.894101    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5354a6-751b-4959-9982-f061b80cf271-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:09.289554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.289514    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf" event={"ID":"0d5354a6-751b-4959-9982-f061b80cf271","Type":"ContainerDied","Data":"feff21ea8e9e53c92023800b757f71f58216588da59c31d2975a98a6cb3fa31b"}
Oct 30 20:09:09.289850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.289830    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feff21ea8e9e53c92023800b757f71f58216588da59c31d2975a98a6cb3fa31b"
Oct 30 20:09:09.296950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.296927    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"
Oct 30 20:09:09.460286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.460253    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:09.461411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.461378    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:09.769795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.766043    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:09.812792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.811886    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:09:09.834795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.832644    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:09.854156 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:09:09.854097    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:09:09.873932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.873890    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:09.953483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:09.953442    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.066883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.066762    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.252726 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.252686    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.273417 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.273366    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.318114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.318008    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.355935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.355881    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.356602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.356564    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.357054 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.357028    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.357456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.357433    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.437785 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.437735    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:10.438053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.438028    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:10.512299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.510866    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.512299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.511213    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.685858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.679858    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.701338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.700885    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.701338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.701042    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 30 20:09:10.768795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.767990    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.770089 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.769310    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:09:10.815794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.814853    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.815794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.815310    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:09:10.836024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.833860    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.836024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.834297    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:09:10.901632 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.875505    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.909924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:10.909888    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:09:11.240860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.048762    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:09:11.240860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.048856    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.403894 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.402200    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.508345 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.508220    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.525505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.508553    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.530964 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.508593    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:09:11.531643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.531621    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:09:11.531884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.510156    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.532033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.525166    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:11.532171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.508633    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.571980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.571937    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.615799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.615740    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.681795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.620852    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:09:11.681795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.680829    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:09:11.681795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.680875    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:09:11.681795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.680915    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:09:11.681795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.680952    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:09:11.681795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.637723    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:11.695845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.653548    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.697987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.657180    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded"
Oct 30 20:09:11.698324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.698294    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:09:11.698875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.680440    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.713835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.660755    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:09:11.713835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.703420    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.713835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.703512    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:09:11.713835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.703876    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.715975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.715951    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:09:11.770472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.770358    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.825796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.821206    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.839502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.839447    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.881447 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:11.881342    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:09:12.030555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.025372    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:09:12.030555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.025432    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.052275 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.052166    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.166275 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.166242    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:09:12.295131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.294885    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:09:12.295131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.294986    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.313091 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.313058    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:09:12.474801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.442336    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.474801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.443339    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:12.474801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.443380    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:12.529786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.517862    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.587852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.566887    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.655889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.654889    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.655889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.655286    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:09:12.655889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.655574    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:09:12.697923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.697887    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:09:12.741171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.739411    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:09:12.741171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.739480    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.743021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.741844    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.821169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.820946    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.924923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.857870    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:12.924923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:12.882915    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:09:13.006702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.006656    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Oct 30 20:09:13.051546 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.051497    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:13.084817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.067312    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:13.084817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.067451    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:09:13.095785 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.095743    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:09:13.116836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.112502    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:09:13.117032 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.116841    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:09:13.145786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.137204    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:09:13.155532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.155488    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:09:13.158952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.158933    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:09:13.168674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.168648    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:09:13.197813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.197791    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:09:13.238824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.238449    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:09:13.241002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.239755    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:09:13.242021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.241222    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:09:13.437095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.437045    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:09:13.437287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:13.437104    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 20:09:14.545113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:14.545076    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:14.545630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:14.545539    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:14.546382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:14.546361    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:09:16.062104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:16.062069    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"]
Oct 30 20:09:16.067402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:16.067373    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d1bd1-6x8l6"]
Oct 30 20:09:18.006038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:18.005997    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c335b54-1675-4358-8cf9-44e7a07a7287" path="/var/lib/kubelet/pods/1c335b54-1675-4358-8cf9-44e7a07a7287/volumes"
Oct 30 20:09:18.107108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:18.107077    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"]
Oct 30 20:09:18.157599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:18.157563    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed2a950-lkn42"]
Oct 30 20:09:20.015380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:20.015342    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f001b444-9423-49cb-ae7f-a9d33ed9bca4" path="/var/lib/kubelet/pods/f001b444-9423-49cb-ae7f-a9d33ed9bca4/volumes"
Oct 30 20:09:27.524548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.524473    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"]
Oct 30 20:09:27.525155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.525134    4917 cpu_manager.go:414] "containerMap: removing container" podUID="0d5354a6-751b-4959-9982-f061b80cf271" containerName="revoke-keys"
Oct 30 20:09:27.525155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.525151    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5354a6-751b-4959-9982-f061b80cf271" containerName="revoke-keys"
Oct 30 20:09:27.525272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.525263    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5354a6-751b-4959-9982-f061b80cf271" containerName="revoke-keys"
Oct 30 20:09:27.530246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.530229    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.586497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.586461    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"]
Oct 30 20:09:27.629497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.629457    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:27.629497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.629492    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:09:27.629708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.629514    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:09:27.635308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.635282    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:09:27.686074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.686033    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.686273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.686139    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf99d\" (UniqueName: \"kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.686273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.686253    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.788076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.787979    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.788076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.788052    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf99d\" (UniqueName: \"kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.788393 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.788139    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.788994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.788965    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.790968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.790940    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:27.851094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:27.851064    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf99d\" (UniqueName: \"kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d\") pod \"maas-api-revoke-sub-bc9d401f-rgws2\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:28.144724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:28.144635    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:28.603105 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:09:28.603070    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3954af2c_3296_4e4b_b1d6_b8bd75ae3073.slice/crio-1b9b7ef12a85feb057d183383d7f544bb3f44767031efaa0f40a697067fa91ad WatchSource:0}: Error finding container 1b9b7ef12a85feb057d183383d7f544bb3f44767031efaa0f40a697067fa91ad: Status 404 returned error can't find the container with id 1b9b7ef12a85feb057d183383d7f544bb3f44767031efaa0f40a697067fa91ad
Oct 30 20:09:28.606456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:28.606434    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"]
Oct 30 20:09:29.088186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:29.088147    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"]
Oct 30 20:09:29.135048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:29.135013    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aafd23a-4dqwp"]
Oct 30 20:09:29.294901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:29.294682    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2" event={"ID":"3954af2c-3296-4e4b-b1d6-b8bd75ae3073","Type":"ContainerStarted","Data":"436f884326ecfefdae510a44fd00d24cdc54cee6970ceb026d75924808b13126"}
Oct 30 20:09:29.294901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:29.294824    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2" event={"ID":"3954af2c-3296-4e4b-b1d6-b8bd75ae3073","Type":"ContainerStarted","Data":"1b9b7ef12a85feb057d183383d7f544bb3f44767031efaa0f40a697067fa91ad"}
Oct 30 20:09:29.416280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:29.416218    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2" podStartSLOduration=2.4162055049999998 podStartE2EDuration="2.416205505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:29.415002052 +0000 UTC m=+1665.579403696" watchObservedRunningTime="2026-10-30 20:09:29.416205505 +0000 UTC m=+1665.580607146"
Oct 30 20:09:30.003496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:30.003421    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d899cdf-dd9d-4182-a6b6-405040315216" path="/var/lib/kubelet/pods/5d899cdf-dd9d-4182-a6b6-405040315216/volumes"
Oct 30 20:09:30.303916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:30.303835    4917 generic.go:356] "Generic (PLEG): container finished" podID="3954af2c-3296-4e4b-b1d6-b8bd75ae3073" containerID="436f884326ecfefdae510a44fd00d24cdc54cee6970ceb026d75924808b13126" exitCode=0
Oct 30 20:09:30.304069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:30.303920    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2" event={"ID":"3954af2c-3296-4e4b-b1d6-b8bd75ae3073","Type":"ContainerDied","Data":"436f884326ecfefdae510a44fd00d24cdc54cee6970ceb026d75924808b13126"}
Oct 30 20:09:31.473276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.473255    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:31.545923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.545897    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca\") pod \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") "
Oct 30 20:09:31.546097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.545960    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d\" (UniqueName: \"kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d\") pod \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") "
Oct 30 20:09:31.546097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.546088    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca\") pod \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\" (UID: \"3954af2c-3296-4e4b-b1d6-b8bd75ae3073\") "
Oct 30 20:09:31.546293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.546263    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca" pod "3954af2c-3296-4e4b-b1d6-b8bd75ae3073" (UID: "3954af2c-3296-4e4b-b1d6-b8bd75ae3073"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:31.546694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.546670    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:31.548034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.548010    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca" pod "3954af2c-3296-4e4b-b1d6-b8bd75ae3073" (UID: "3954af2c-3296-4e4b-b1d6-b8bd75ae3073"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:31.548092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.548053    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d" pod "3954af2c-3296-4e4b-b1d6-b8bd75ae3073" (UID: "3954af2c-3296-4e4b-b1d6-b8bd75ae3073"). InnerVolumeSpecName "kube-api-access-nf99d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:31.648124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.648042    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf99d\" (UniqueName: \"kubernetes.io/projected/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-kube-api-access-nf99d\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:31.648124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:31.648070    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3954af2c-3296-4e4b-b1d6-b8bd75ae3073-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:32.320587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:32.320515    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"
Oct 30 20:09:32.320587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:32.320553    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2" event={"ID":"3954af2c-3296-4e4b-b1d6-b8bd75ae3073","Type":"ContainerDied","Data":"1b9b7ef12a85feb057d183383d7f544bb3f44767031efaa0f40a697067fa91ad"}
Oct 30 20:09:32.320805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:32.320623    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9b7ef12a85feb057d183383d7f544bb3f44767031efaa0f40a697067fa91ad"
Oct 30 20:09:33.676302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.676265    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"]
Oct 30 20:09:33.676947 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.676931    4917 cpu_manager.go:414] "containerMap: removing container" podUID="3954af2c-3296-4e4b-b1d6-b8bd75ae3073" containerName="revoke-keys"
Oct 30 20:09:33.677010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.676949    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3954af2c-3296-4e4b-b1d6-b8bd75ae3073" containerName="revoke-keys"
Oct 30 20:09:33.677098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.677089    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="3954af2c-3296-4e4b-b1d6-b8bd75ae3073" containerName="revoke-keys"
Oct 30 20:09:33.680893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.680878    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.738968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.738932    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"]
Oct 30 20:09:33.775033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.774994    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbkb\" (UniqueName: \"kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.775306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.775285    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.775408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.775343    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.784516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.784490    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:09:33.784656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.784527    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:09:33.784656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.784527    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:33.784656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.784572    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:09:33.876783 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.876750    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.876957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.876814    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.876957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.876866    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbkb\" (UniqueName: \"kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.877497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.877476    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.879212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.879192    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.939423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.939397    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbkb\" (UniqueName: \"kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb\") pod \"maas-api-revoke-sub-084205fb-zwh8z\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:33.995533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:33.995498    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:34.386741 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:09:34.386649    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd5a668_2bb3_420d_b512_9b549717b9a2.slice/crio-da1ec5937c86a010dd3c9811665b276bbea756071ce2f87a43bce0c0ebd35ef8 WatchSource:0}: Error finding container da1ec5937c86a010dd3c9811665b276bbea756071ce2f87a43bce0c0ebd35ef8: Status 404 returned error can't find the container with id da1ec5937c86a010dd3c9811665b276bbea756071ce2f87a43bce0c0ebd35ef8
Oct 30 20:09:34.388441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:34.388421    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"]
Oct 30 20:09:35.347102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:35.347067    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z" event={"ID":"fbd5a668-2bb3-420d-b512-9b549717b9a2","Type":"ContainerStarted","Data":"4dfc362c48953ac624441ae73788321b5d42a79bc4cb97a10be40a440bbdf7e4"}
Oct 30 20:09:35.347509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:35.347122    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z" event={"ID":"fbd5a668-2bb3-420d-b512-9b549717b9a2","Type":"ContainerStarted","Data":"da1ec5937c86a010dd3c9811665b276bbea756071ce2f87a43bce0c0ebd35ef8"}
Oct 30 20:09:35.465683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:35.465630    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z" podStartSLOduration=2.465616439 podStartE2EDuration="2.465616439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:35.464332282 +0000 UTC m=+1671.628733929" watchObservedRunningTime="2026-10-30 20:09:35.465616439 +0000 UTC m=+1671.630018102"
Oct 30 20:09:36.356863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:36.356818    4917 generic.go:356] "Generic (PLEG): container finished" podID="fbd5a668-2bb3-420d-b512-9b549717b9a2" containerID="4dfc362c48953ac624441ae73788321b5d42a79bc4cb97a10be40a440bbdf7e4" exitCode=0
Oct 30 20:09:36.357259 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:36.356893    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z" event={"ID":"fbd5a668-2bb3-420d-b512-9b549717b9a2","Type":"ContainerDied","Data":"4dfc362c48953ac624441ae73788321b5d42a79bc4cb97a10be40a440bbdf7e4"}
Oct 30 20:09:37.536058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.536030    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:37.633472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.633379    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca\") pod \"fbd5a668-2bb3-420d-b512-9b549717b9a2\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") "
Oct 30 20:09:37.633644 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.633533    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca\") pod \"fbd5a668-2bb3-420d-b512-9b549717b9a2\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") "
Oct 30 20:09:37.633711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.633665    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb\" (UniqueName: \"kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb\") pod \"fbd5a668-2bb3-420d-b512-9b549717b9a2\" (UID: \"fbd5a668-2bb3-420d-b512-9b549717b9a2\") "
Oct 30 20:09:37.633934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.633908    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca" pod "fbd5a668-2bb3-420d-b512-9b549717b9a2" (UID: "fbd5a668-2bb3-420d-b512-9b549717b9a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:37.634273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.634252    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:37.635490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.635470    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca" pod "fbd5a668-2bb3-420d-b512-9b549717b9a2" (UID: "fbd5a668-2bb3-420d-b512-9b549717b9a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:37.635672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.635650    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb" pod "fbd5a668-2bb3-420d-b512-9b549717b9a2" (UID: "fbd5a668-2bb3-420d-b512-9b549717b9a2"). InnerVolumeSpecName "kube-api-access-wlbkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:37.736429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.736384    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbkb\" (UniqueName: \"kubernetes.io/projected/fbd5a668-2bb3-420d-b512-9b549717b9a2-kube-api-access-wlbkb\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:37.736429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:37.736430    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd5a668-2bb3-420d-b512-9b549717b9a2-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:38.330753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.330666    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"]
Oct 30 20:09:38.331428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.331410    4917 cpu_manager.go:414] "containerMap: removing container" podUID="fbd5a668-2bb3-420d-b512-9b549717b9a2" containerName="revoke-keys"
Oct 30 20:09:38.331515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.331429    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd5a668-2bb3-420d-b512-9b549717b9a2" containerName="revoke-keys"
Oct 30 20:09:38.331560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.331554    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd5a668-2bb3-420d-b512-9b549717b9a2" containerName="revoke-keys"
Oct 30 20:09:38.335384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.335363    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.387892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.383216    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z" event={"ID":"fbd5a668-2bb3-420d-b512-9b549717b9a2","Type":"ContainerDied","Data":"da1ec5937c86a010dd3c9811665b276bbea756071ce2f87a43bce0c0ebd35ef8"}
Oct 30 20:09:38.387892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.383285    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da1ec5937c86a010dd3c9811665b276bbea756071ce2f87a43bce0c0ebd35ef8"
Oct 30 20:09:38.387892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.383380    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"
Oct 30 20:09:38.398168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.398132    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d7ca5342-dockercfg-76czr\""
Oct 30 20:09:38.406147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.406118    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d7ca5342\""
Oct 30 20:09:38.409331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.409288    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"]
Oct 30 20:09:38.448209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.448166    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9v5\" (UniqueName: \"kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5\") pod \"payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.448633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.448259    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.552826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.550048    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9v5\" (UniqueName: \"kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5\") pod \"payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.552826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.550141    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.552826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.551201    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.613093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.613003    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9v5\" (UniqueName: \"kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5\") pod \"payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:38.649584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:38.649548    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:39.057420 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:09:39.057374    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a2ac90_0da7_48a0_bfce_49d42493ea94.slice/crio-35eea8f6c8cd13537c2a0e2b894b7d18dca40c703b76ceb7c03ec29042bfd57a WatchSource:0}: Error finding container 35eea8f6c8cd13537c2a0e2b894b7d18dca40c703b76ceb7c03ec29042bfd57a: Status 404 returned error can't find the container with id 35eea8f6c8cd13537c2a0e2b894b7d18dca40c703b76ceb7c03ec29042bfd57a
Oct 30 20:09:39.058313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:39.058289    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"]
Oct 30 20:09:39.394402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:39.394312    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx" event={"ID":"b3a2ac90-0da7-48a0-bfce-49d42493ea94","Type":"ContainerStarted","Data":"4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0"}
Oct 30 20:09:39.394402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:39.394377    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx" event={"ID":"b3a2ac90-0da7-48a0-bfce-49d42493ea94","Type":"ContainerStarted","Data":"35eea8f6c8cd13537c2a0e2b894b7d18dca40c703b76ceb7c03ec29042bfd57a"}
Oct 30 20:09:39.394591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:39.394408    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:39.518379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:39.518304    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx" podStartSLOduration=1.5182795439999999 podStartE2EDuration="1.518279544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:39.513060357 +0000 UTC m=+1675.677461998" watchObservedRunningTime="2026-10-30 20:09:39.518279544 +0000 UTC m=+1675.682681187"
Oct 30 20:09:39.681190 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:09:39.681149    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:09:44.497875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.497841    4917 scope.go:122] "RemoveContainer" containerID="64b9f1d08727addd32f5b0d20688ae5e2ce3d32ce0bda185a5ee37a39bb97093"
Oct 30 20:09:44.511577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.511549    4917 scope.go:122] "RemoveContainer" containerID="c0bc262f6c40210bd0d431e6b48c978c48673f629e02fae83513f5ac4b9b472c"
Oct 30 20:09:44.570095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.570057    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"]
Oct 30 20:09:44.576827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.576804    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.583266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583237    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.583434 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583413    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.583508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583464    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8h4\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.583566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583557    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.583622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583604    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.583723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583705    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.584632 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.583764    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.631972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.631932    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:09:44.632150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.631986    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:09:44.632150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.632010    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:09:44.682162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.682122    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jf4m5\""
Oct 30 20:09:44.686217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.686178    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.686484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.686460    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.686634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.686614    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.686848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.686829    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.686986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.686970    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.687217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.687196    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.687376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.687359    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8h4\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.689029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.689003    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"]
Oct 30 20:09:44.689686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.689626    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.691396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.690490    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.691396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.691289    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.692168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.692148    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.697244 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.697222    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.750103 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.750034    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8h4\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.750250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.750226    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lh2q6\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:44.894301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:44.894272    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:45.358192 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:09:45.358161    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba833ad_696a_47a2_acb7_7b8c8aa84278.slice/crio-81db0796fcb5f8ee996ffc04a0cd9daf81e70ec813b9359ac048ecfc6c5e1db9 WatchSource:0}: Error finding container 81db0796fcb5f8ee996ffc04a0cd9daf81e70ec813b9359ac048ecfc6c5e1db9: Status 404 returned error can't find the container with id 81db0796fcb5f8ee996ffc04a0cd9daf81e70ec813b9359ac048ecfc6c5e1db9
Oct 30 20:09:45.360664 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:45.360645    4917 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:09:45.361608 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:45.361589    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"]
Oct 30 20:09:45.445765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:45.445728    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" event={"ID":"6ba833ad-696a-47a2-acb7-7b8c8aa84278","Type":"ContainerStarted","Data":"81db0796fcb5f8ee996ffc04a0cd9daf81e70ec813b9359ac048ecfc6c5e1db9"}
Oct 30 20:09:48.757796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:48.756894    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:48.757796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:48.757570    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:48.890913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:48.890864    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.088030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.086635    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.107928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.107074    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.128017 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.127962    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.133121 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.133064    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.181720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.181677    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.252961 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.252907    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.253668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.253631    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.456938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.456891    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.570805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.570331    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.602860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.601625    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.933903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.933857    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.934426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.934301    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.934714 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.934684    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:09:49.953824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.933635    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.953824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.935673    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:09:49.953824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.936194    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.953824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.936510    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:09:50.066836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.050301    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 20:09:50.127793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.127181    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.143303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.141212    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.143303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.141586    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.143303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.142624    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.212422 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.212322    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.254798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.253928    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.261284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.260844    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.317797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.294846    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 20:09:50.470852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:49.935337    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.501795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.472837    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.501795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.496324    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.552184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.552152    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605283    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605363    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605399    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605436    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605467    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605498    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605534    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605571    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:09:50.605794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.605616    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:09:50.632871 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.609847    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.632871 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.610131    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:09:50.752668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.750815    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.962911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.958097    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.974110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.974060    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.988284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.988226    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.991848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:50.991801    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.008840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.008728    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:09:51.187175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.187120    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.188746 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.188711    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:09:51.191718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.191665    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:09:51.232220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.232175    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:09:51.277616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.276561    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.277616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.276660    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Oct 30 20:09:51.294143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.292878    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.601850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.601121    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.601850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.601552    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:09:51.602496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.602461    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.618805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.617934    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.962180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.959265    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.988818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:51.985361    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.007558 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.003543    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.007558 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.003922    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.055585 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.051881    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.225850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.223228    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.247409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.245741    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.247409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.246130    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.247409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.246873    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.247741 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.247713    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.285310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.285269    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.305104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.305059    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.445388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.444950    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.701858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.680938    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.701858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.682792    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.701858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.661284    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.900809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.897113    4917 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 20:09:52.960820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:52.960683    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.071885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.022854    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.071885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.051897    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.071885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.052284    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.227966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.227859    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.244391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.243143    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:09:53.288016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.287975    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.289281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.289247    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.290499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.290474    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.296229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.296209    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:09:53.339999 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.338866    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.342875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.342828    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.357795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.346604    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.649934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.649854    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:09:53.683990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.683231    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.691723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.684258    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.761520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.760859    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.795811 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.794748    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:09:53.877247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:53.874953    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:09:54.080984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.080957    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081014    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081037    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081060    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081091    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081123    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081144    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081172    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081201    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:09:54.081391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:54.081224    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:09:56.137233 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:56.137197    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:09:56.137619 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:56.137267    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:09:56.430646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:56.430605    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" event={"ID":"6ba833ad-696a-47a2-acb7-7b8c8aa84278","Type":"ContainerStarted","Data":"d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d"}
Oct 30 20:09:56.430830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:56.430664    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:09:56.608107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:56.608042    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" podStartSLOduration=1.833029886 podStartE2EDuration="12.608027726s" totalImagesPullingTime="10.77499784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:56.607089681 +0000 UTC m=+1692.771491324" watchObservedRunningTime="2026-10-30 20:09:56.608027726 +0000 UTC m=+1692.772429368"
Oct 30 20:09:57.444505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:57.444472    4917 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lh2q6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:09:57.444936 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:09:57.444528    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" podUID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:00.443248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:00.443217    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:10:08.826276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.826233    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"]
Oct 30 20:10:08.831144 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.831112    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.970843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.970808    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.970865    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.970901    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvrr\" (UniqueName: \"kubernetes.io/projected/b17447df-4223-406e-8271-9c6d9b5e15a6-kube-api-access-4zvrr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971170 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.971076    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971170 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.971104    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971170 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.971126    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.971225    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.971254    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17447df-4223-406e-8271-9c6d9b5e15a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:08.971524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:08.971377    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.043949 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.043915    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"]
Oct 30 20:10:09.073170 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073136    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073441    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073490    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvrr\" (UniqueName: \"kubernetes.io/projected/b17447df-4223-406e-8271-9c6d9b5e15a6-kube-api-access-4zvrr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073549    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073578    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073605    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073691    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073737    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17447df-4223-406e-8271-9c6d9b5e15a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.073972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.073866    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.074384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.074362    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.074925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.074900    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.075192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.075169    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.075419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.075397    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.076118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.076039    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17447df-4223-406e-8271-9c6d9b5e15a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.076422 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.076123    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.076981 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.076956    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.136698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.136660    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvrr\" (UniqueName: \"kubernetes.io/projected/b17447df-4223-406e-8271-9c6d9b5e15a6-kube-api-access-4zvrr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.136876 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.136730    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17447df-4223-406e-8271-9c6d9b5e15a6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-kt69c\" (UID: \"b17447df-4223-406e-8271-9c6d9b5e15a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.145885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.145862    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:09.606646 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:10:09.606606    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17447df_4223_406e_8271_9c6d9b5e15a6.slice/crio-6eee44b52e70a0457452df3684c961a637294ff6f75dde64e2c4ee312448ca93 WatchSource:0}: Error finding container 6eee44b52e70a0457452df3684c961a637294ff6f75dde64e2c4ee312448ca93: Status 404 returned error can't find the container with id 6eee44b52e70a0457452df3684c961a637294ff6f75dde64e2c4ee312448ca93
Oct 30 20:10:09.606879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.606856    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"]
Oct 30 20:10:09.609565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.609534    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:10:09.609685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.609612    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:10:09.609685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:09.609656    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:10:09.684211 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:10:09.684171    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:10:10.573528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:10.573490    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" event={"ID":"b17447df-4223-406e-8271-9c6d9b5e15a6","Type":"ContainerStarted","Data":"c082c3df665bf53e8b498b8cc9c943d1759bde7966f6b213c5dd2c8c831b8dd3"}
Oct 30 20:10:10.573985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:10.573547    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" event={"ID":"b17447df-4223-406e-8271-9c6d9b5e15a6","Type":"ContainerStarted","Data":"6eee44b52e70a0457452df3684c961a637294ff6f75dde64e2c4ee312448ca93"}
Oct 30 20:10:10.824755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:10.824650    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podStartSLOduration=2.824634975 podStartE2EDuration="2.824634975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:10.777392616 +0000 UTC m=+1706.941794258" watchObservedRunningTime="2026-10-30 20:10:10.824634975 +0000 UTC m=+1706.989036617"
Oct 30 20:10:11.158850 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:11.157157    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:11.169798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:11.164966    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:11.169798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:11.165051    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:12.153800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:12.148127    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:12.153800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:12.148192    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:13.148793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:13.148145    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:13.148793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:13.148225    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:14.155643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:14.155186    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:14.155643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:14.155248    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:15.152159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:15.152119    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:15.152496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:15.152467    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:16.146223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:16.146180    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:16.146740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:16.146238    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:17.147227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:17.147190    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:10:17.147634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:17.147246    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 20:10:18.532494 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:18.532458    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:18.533009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:18.532970    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:18.533702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:18.533682    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c"
Oct 30 20:10:23.432725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:23.432690    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:10:37.056766 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:37.056732    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"]
Oct 30 20:10:37.106346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:37.106315    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4439202-sqvqp"]
Oct 30 20:10:38.003857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:38.003824    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba" path="/var/lib/kubelet/pods/a28cb1ba-dbf1-4a7d-99e3-ac70252dfcba/volumes"
Oct 30 20:10:39.681831 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:10:39.681797    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:10:44.549010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:44.548977    4917 scope.go:122] "RemoveContainer" containerID="01700c11650e4dccd4f88b168801a2b9f881786eec2f05a3b4d4884364b9307a"
Oct 30 20:10:44.560388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:44.560362    4917 scope.go:122] "RemoveContainer" containerID="b1b1cd73bc3c9df82e3e1d9ebed1123f23781fb991a1a4be1a392143c1fa4478"
Oct 30 20:10:44.573041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:10:44.573021    4917 scope.go:122] "RemoveContainer" containerID="887b23b0c0b5ebebe62f76f5f75cf28d6fd755714e153376ca672ba253678ceb"
Oct 30 20:11:01.055183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:01.055142    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"]
Oct 30 20:11:01.104891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:01.104855    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79af6e11-bzmwz"]
Oct 30 20:11:02.005011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:02.004981    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e30dbf2-2c4d-43ff-a585-676c520f40ee" path="/var/lib/kubelet/pods/0e30dbf2-2c4d-43ff-a585-676c520f40ee/volumes"
Oct 30 20:11:08.973676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:08.973639    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"]
Oct 30 20:11:08.978219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:08.978196    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.021036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.021000    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.021186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.021078    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.021186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.021172    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzpn\" (UniqueName: \"kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.036224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.036200    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"]
Oct 30 20:11:09.080539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.080504    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:09.080686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.080558    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:11:09.082211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.082195    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:11:09.082291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.082280    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:11:09.122743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.122664    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.122743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.122735    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.123015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.122798    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzpn\" (UniqueName: \"kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.123515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.123494    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.125669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.125644    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.184892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.184860    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzpn\" (UniqueName: \"kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn\") pod \"maas-api-revoke-sub-dcf6c4eb-6mccn\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.292446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.292417    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:09.676931 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:11:09.676896    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b94266_b4f6_4199_96ce_c6bca2396bd5.slice/crio-f71741994b9109f80b69b0aabe15c60cb92a1dfb27dd6076b4f75292cdf38ce8 WatchSource:0}: Error finding container f71741994b9109f80b69b0aabe15c60cb92a1dfb27dd6076b4f75292cdf38ce8: Status 404 returned error can't find the container with id f71741994b9109f80b69b0aabe15c60cb92a1dfb27dd6076b4f75292cdf38ce8
Oct 30 20:11:09.679381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:09.679202    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"]
Oct 30 20:11:09.685998 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:11:09.685970    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:11:10.115113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:10.115070    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn" event={"ID":"a8b94266-b4f6-4199-96ce-c6bca2396bd5","Type":"ContainerStarted","Data":"d346c2c11204cdc0ed4c2815e338a15014fb9560f878d964cee39c6220af1dd0"}
Oct 30 20:11:10.115516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:10.115141    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn" event={"ID":"a8b94266-b4f6-4199-96ce-c6bca2396bd5","Type":"ContainerStarted","Data":"f71741994b9109f80b69b0aabe15c60cb92a1dfb27dd6076b4f75292cdf38ce8"}
Oct 30 20:11:10.237368 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:10.237314    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn" podStartSLOduration=2.237298038 podStartE2EDuration="2.237298038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:10.23450388 +0000 UTC m=+1766.398905522" watchObservedRunningTime="2026-10-30 20:11:10.237298038 +0000 UTC m=+1766.401699680"
Oct 30 20:11:11.125309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:11.125273    4917 generic.go:356] "Generic (PLEG): container finished" podID="a8b94266-b4f6-4199-96ce-c6bca2396bd5" containerID="d346c2c11204cdc0ed4c2815e338a15014fb9560f878d964cee39c6220af1dd0" exitCode=0
Oct 30 20:11:11.125733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:11.125373    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn" event={"ID":"a8b94266-b4f6-4199-96ce-c6bca2396bd5","Type":"ContainerDied","Data":"d346c2c11204cdc0ed4c2815e338a15014fb9560f878d964cee39c6220af1dd0"}
Oct 30 20:11:12.071806 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.071745    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"]
Oct 30 20:11:12.121502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.121469    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71263e87-kk978"]
Oct 30 20:11:12.312814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.312787    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:12.368587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.368512    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn\" (UniqueName: \"kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn\") pod \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") "
Oct 30 20:11:12.368730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.368616    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca\") pod \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") "
Oct 30 20:11:12.368879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.368846    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca\") pod \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\" (UID: \"a8b94266-b4f6-4199-96ce-c6bca2396bd5\") "
Oct 30 20:11:12.369169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.369148    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca" pod "a8b94266-b4f6-4199-96ce-c6bca2396bd5" (UID: "a8b94266-b4f6-4199-96ce-c6bca2396bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:12.369490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.369473    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:12.370750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.370723    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca" pod "a8b94266-b4f6-4199-96ce-c6bca2396bd5" (UID: "a8b94266-b4f6-4199-96ce-c6bca2396bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:12.370872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.370745    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn" pod "a8b94266-b4f6-4199-96ce-c6bca2396bd5" (UID: "a8b94266-b4f6-4199-96ce-c6bca2396bd5"). InnerVolumeSpecName "kube-api-access-bpzpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:12.470515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.470478    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzpn\" (UniqueName: \"kubernetes.io/projected/a8b94266-b4f6-4199-96ce-c6bca2396bd5-kube-api-access-bpzpn\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:12.470515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:12.470516    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b94266-b4f6-4199-96ce-c6bca2396bd5-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.147033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:13.146992    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn" event={"ID":"a8b94266-b4f6-4199-96ce-c6bca2396bd5","Type":"ContainerDied","Data":"f71741994b9109f80b69b0aabe15c60cb92a1dfb27dd6076b4f75292cdf38ce8"}
Oct 30 20:11:13.147235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:13.147064    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f71741994b9109f80b69b0aabe15c60cb92a1dfb27dd6076b4f75292cdf38ce8"
Oct 30 20:11:13.147235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:13.147009    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"
Oct 30 20:11:14.006728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:14.006696    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dac310-e527-424b-8e3b-1cb4d3527872" path="/var/lib/kubelet/pods/28dac310-e527-424b-8e3b-1cb4d3527872/volumes"
Oct 30 20:11:18.990615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:18.990579    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"]
Oct 30 20:11:18.991311 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:18.991294    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a8b94266-b4f6-4199-96ce-c6bca2396bd5" containerName="revoke-keys"
Oct 30 20:11:18.991369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:18.991312    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b94266-b4f6-4199-96ce-c6bca2396bd5" containerName="revoke-keys"
Oct 30 20:11:18.991484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:18.991473    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b94266-b4f6-4199-96ce-c6bca2396bd5" containerName="revoke-keys"
Oct 30 20:11:18.995658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:18.995629    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.051209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.049567    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:19.051209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.050101    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:11:19.051209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.050143    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:11:19.055280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.055255    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"]
Oct 30 20:11:19.069925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.069898    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7wt\" (UniqueName: \"kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.070067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.070021    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.070266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.070241    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.098832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.098799    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:11:19.172316 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.172280    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.172482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.172448    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7wt\" (UniqueName: \"kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.172571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.172496    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.173248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.173222    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.174749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.174727    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.235332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.235296    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7wt\" (UniqueName: \"kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt\") pod \"maas-api-revoke-keys-e2e-migrate-27915880-s9v7n\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.310790 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.310691    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:19.698760 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:19.698730    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"]
Oct 30 20:11:19.698889 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:11:19.698739    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b947c0_480d_43b5_87f3_daa484266f27.slice/crio-7993399d9a299383399547623a15050680e0ea68778fef94484093014e7ad6a8 WatchSource:0}: Error finding container 7993399d9a299383399547623a15050680e0ea68778fef94484093014e7ad6a8: Status 404 returned error can't find the container with id 7993399d9a299383399547623a15050680e0ea68778fef94484093014e7ad6a8
Oct 30 20:11:20.208814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:20.208697    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n" event={"ID":"24b947c0-480d-43b5-87f3-daa484266f27","Type":"ContainerStarted","Data":"69cb70e4dbed48df91f78c0f6010eaf8611b527296b91ca29dc912e0a0959101"}
Oct 30 20:11:20.208814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:20.208751    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n" event={"ID":"24b947c0-480d-43b5-87f3-daa484266f27","Type":"ContainerStarted","Data":"7993399d9a299383399547623a15050680e0ea68778fef94484093014e7ad6a8"}
Oct 30 20:11:20.330697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:20.330637    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n" podStartSLOduration=2.330621758 podStartE2EDuration="2.330621758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:20.328691075 +0000 UTC m=+1776.493092718" watchObservedRunningTime="2026-10-30 20:11:20.330621758 +0000 UTC m=+1776.495023451"
Oct 30 20:11:21.056875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:21.056837    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"]
Oct 30 20:11:21.105062 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:21.105032    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e323603-gdzs9"]
Oct 30 20:11:21.218719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:21.218686    4917 generic.go:356] "Generic (PLEG): container finished" podID="24b947c0-480d-43b5-87f3-daa484266f27" containerID="69cb70e4dbed48df91f78c0f6010eaf8611b527296b91ca29dc912e0a0959101" exitCode=0
Oct 30 20:11:21.219119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:21.218790    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n" event={"ID":"24b947c0-480d-43b5-87f3-daa484266f27","Type":"ContainerDied","Data":"69cb70e4dbed48df91f78c0f6010eaf8611b527296b91ca29dc912e0a0959101"}
Oct 30 20:11:22.004596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.004562    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9c91d7-abb9-4731-a42c-2c7961a1de9a" path="/var/lib/kubelet/pods/1f9c91d7-abb9-4731-a42c-2c7961a1de9a/volumes"
Oct 30 20:11:22.396729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.396696    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:22.518730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.518693    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca\") pod \"24b947c0-480d-43b5-87f3-daa484266f27\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") "
Oct 30 20:11:22.519029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.518995    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca\") pod \"24b947c0-480d-43b5-87f3-daa484266f27\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") "
Oct 30 20:11:22.519199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.519058    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt\" (UniqueName: \"kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt\") pod \"24b947c0-480d-43b5-87f3-daa484266f27\" (UID: \"24b947c0-480d-43b5-87f3-daa484266f27\") "
Oct 30 20:11:22.520937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.519580    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca" pod "24b947c0-480d-43b5-87f3-daa484266f27" (UID: "24b947c0-480d-43b5-87f3-daa484266f27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:22.520937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.519937    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.525128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.525098    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca" pod "24b947c0-480d-43b5-87f3-daa484266f27" (UID: "24b947c0-480d-43b5-87f3-daa484266f27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:22.525280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.525219    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt" pod "24b947c0-480d-43b5-87f3-daa484266f27" (UID: "24b947c0-480d-43b5-87f3-daa484266f27"). InnerVolumeSpecName "kube-api-access-ph7wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:22.621357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.621260    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7wt\" (UniqueName: \"kubernetes.io/projected/24b947c0-480d-43b5-87f3-daa484266f27-kube-api-access-ph7wt\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.621357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:22.621304    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b947c0-480d-43b5-87f3-daa484266f27-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.239948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:23.239915    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"
Oct 30 20:11:23.240139 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:23.239943    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n" event={"ID":"24b947c0-480d-43b5-87f3-daa484266f27","Type":"ContainerDied","Data":"7993399d9a299383399547623a15050680e0ea68778fef94484093014e7ad6a8"}
Oct 30 20:11:23.240139 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:23.239992    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7993399d9a299383399547623a15050680e0ea68778fef94484093014e7ad6a8"
Oct 30 20:11:24.101425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.101390    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"]
Oct 30 20:11:24.102145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.102128    4917 cpu_manager.go:414] "containerMap: removing container" podUID="24b947c0-480d-43b5-87f3-daa484266f27" containerName="revoke-keys"
Oct 30 20:11:24.102205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.102146    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b947c0-480d-43b5-87f3-daa484266f27" containerName="revoke-keys"
Oct 30 20:11:24.102287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.102277    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b947c0-480d-43b5-87f3-daa484266f27" containerName="revoke-keys"
Oct 30 20:11:24.105985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.105965    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.165290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.165260    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"]
Oct 30 20:11:24.208917 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.208886    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:11:24.209124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.208927    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:24.209124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.209032    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:11:24.209301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.209279    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:11:24.246313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.246280    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.246617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.246590    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fv7\" (UniqueName: \"kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.246958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.246931    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.348419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.348375    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fv7\" (UniqueName: \"kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.348590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.348546    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.348713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.348694    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.349514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.349486    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.351045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.351023    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.411528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.411402    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fv7\" (UniqueName: \"kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7\") pod \"maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.421192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.421160    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:24.691030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.690996    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:11:24.807837 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:11:24.807808    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa255090_f05a_4f09_9d3e_81e73a82d827.slice/crio-cbef12a6540c5178082164960014732183e40469dd89adc929436cfdbe212fad WatchSource:0}: Error finding container cbef12a6540c5178082164960014732183e40469dd89adc929436cfdbe212fad: Status 404 returned error can't find the container with id cbef12a6540c5178082164960014732183e40469dd89adc929436cfdbe212fad
Oct 30 20:11:24.810024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:24.809997    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"]
Oct 30 20:11:25.283348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:25.283289    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8" event={"ID":"aa255090-f05a-4f09-9d3e-81e73a82d827","Type":"ContainerStarted","Data":"e1f0eee70f93041a01f3e02f27391265ddb662750aa886efaff8739767ce9f94"}
Oct 30 20:11:25.284207 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:25.284168    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8" event={"ID":"aa255090-f05a-4f09-9d3e-81e73a82d827","Type":"ContainerStarted","Data":"cbef12a6540c5178082164960014732183e40469dd89adc929436cfdbe212fad"}
Oct 30 20:11:25.408500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:25.408389    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8" podStartSLOduration=1.40837337 podStartE2EDuration="1.40837337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:25.40540293 +0000 UTC m=+1781.569804609" watchObservedRunningTime="2026-10-30 20:11:25.40837337 +0000 UTC m=+1781.572775012"
Oct 30 20:11:26.292746 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:26.292705    4917 generic.go:356] "Generic (PLEG): container finished" podID="aa255090-f05a-4f09-9d3e-81e73a82d827" containerID="e1f0eee70f93041a01f3e02f27391265ddb662750aa886efaff8739767ce9f94" exitCode=0
Oct 30 20:11:26.293160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:26.292806    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8" event={"ID":"aa255090-f05a-4f09-9d3e-81e73a82d827","Type":"ContainerDied","Data":"e1f0eee70f93041a01f3e02f27391265ddb662750aa886efaff8739767ce9f94"}
Oct 30 20:11:27.462726 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.462702    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:27.602301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.602218    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca\") pod \"aa255090-f05a-4f09-9d3e-81e73a82d827\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") "
Oct 30 20:11:27.602492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.602471    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca\") pod \"aa255090-f05a-4f09-9d3e-81e73a82d827\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") "
Oct 30 20:11:27.602581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.602547    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca" pod "aa255090-f05a-4f09-9d3e-81e73a82d827" (UID: "aa255090-f05a-4f09-9d3e-81e73a82d827"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:27.602693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.602681    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7\" (UniqueName: \"kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7\") pod \"aa255090-f05a-4f09-9d3e-81e73a82d827\" (UID: \"aa255090-f05a-4f09-9d3e-81e73a82d827\") "
Oct 30 20:11:27.603356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.603332    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.605275 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.605243    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca" pod "aa255090-f05a-4f09-9d3e-81e73a82d827" (UID: "aa255090-f05a-4f09-9d3e-81e73a82d827"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:27.605414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.605396    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7" pod "aa255090-f05a-4f09-9d3e-81e73a82d827" (UID: "aa255090-f05a-4f09-9d3e-81e73a82d827"). InnerVolumeSpecName "kube-api-access-z6fv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:27.705018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.704988    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa255090-f05a-4f09-9d3e-81e73a82d827-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.705018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:27.705016    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6fv7\" (UniqueName: \"kubernetes.io/projected/aa255090-f05a-4f09-9d3e-81e73a82d827-kube-api-access-z6fv7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.314195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:28.314104    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8" event={"ID":"aa255090-f05a-4f09-9d3e-81e73a82d827","Type":"ContainerDied","Data":"cbef12a6540c5178082164960014732183e40469dd89adc929436cfdbe212fad"}
Oct 30 20:11:28.314195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:28.314170    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbef12a6540c5178082164960014732183e40469dd89adc929436cfdbe212fad"
Oct 30 20:11:28.314195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:28.314130    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"
Oct 30 20:11:34.105816 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.105763    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"]
Oct 30 20:11:34.106469 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.106453    4917 cpu_manager.go:414] "containerMap: removing container" podUID="aa255090-f05a-4f09-9d3e-81e73a82d827" containerName="revoke-keys"
Oct 30 20:11:34.106517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.106470    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa255090-f05a-4f09-9d3e-81e73a82d827" containerName="revoke-keys"
Oct 30 20:11:34.106593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.106584    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa255090-f05a-4f09-9d3e-81e73a82d827" containerName="revoke-keys"
Oct 30 20:11:34.110575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.110553    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.173665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.173618    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7961ed-openshift-default-dockercfg-z4vrz\""
Oct 30 20:11:34.219663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.219621    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.219863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.219759    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.219863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.219843    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.219995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.219879    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.220046 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.219998    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.220087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.220048    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.220138 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.220082    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.220193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.220155    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt57\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.220271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.220246    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.222315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.222288    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"]
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.323927    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.323985    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324024    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324151    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324207    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324240    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324326    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt57\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324417    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.324800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.324480    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.338717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.338681    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.355809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.351232    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.356122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.352014    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.356334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.352174    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.356412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.352681    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.410183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.408429    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.410587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.410420    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.430362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.430296    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.432553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.432434    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt57\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57\") pod \"e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") " pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:34.727352 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:34.727308    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:35.174451 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:11:35.174420    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode280fd88_2d98_449d_ab3d_55ad7bf216f8.slice/crio-b6fe70c1c4143cd3fe50a06850fc1ccd0853a2dbc6034222f565cbc92efede43 WatchSource:0}: Error finding container b6fe70c1c4143cd3fe50a06850fc1ccd0853a2dbc6034222f565cbc92efede43: Status 404 returned error can't find the container with id b6fe70c1c4143cd3fe50a06850fc1ccd0853a2dbc6034222f565cbc92efede43
Oct 30 20:11:35.177048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.177020    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"]
Oct 30 20:11:35.177701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.177661    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:11:35.177783 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.177752    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:11:35.177853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.177836    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:11:35.440582 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.440547    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" event={"ID":"e280fd88-2d98-449d-ab3d-55ad7bf216f8","Type":"ContainerStarted","Data":"1365b76ba70512b221b6bed52067904bb3e199c3f0cd90cf026016a4bf25a726"}
Oct 30 20:11:35.440820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.440612    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" event={"ID":"e280fd88-2d98-449d-ab3d-55ad7bf216f8","Type":"ContainerStarted","Data":"b6fe70c1c4143cd3fe50a06850fc1ccd0853a2dbc6034222f565cbc92efede43"}
Oct 30 20:11:35.568360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.568292    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podStartSLOduration=1.56827114 podStartE2EDuration="1.56827114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:35.566917802 +0000 UTC m=+1791.731319443" watchObservedRunningTime="2026-10-30 20:11:35.56827114 +0000 UTC m=+1791.732672780"
Oct 30 20:11:35.727884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:35.727755    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:36.060534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:36.060436    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"]
Oct 30 20:11:36.068500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:36.068463    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d76a077-x6pgv"]
Oct 30 20:11:36.735436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:36.735404    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:37.385836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.385801    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"]
Oct 30 20:11:37.390363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.390340    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.445982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.445944    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:11:37.446535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.446505    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:11:37.446684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.446614    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:11:37.446684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.446647    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:37.449379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.449346    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"]
Oct 30 20:11:37.459559 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.459532    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:37.460616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.460594    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:11:37.475373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.475350    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.475520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.475501    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v67m\" (UniqueName: \"kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.475705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.475692    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.577043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.576993    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v67m\" (UniqueName: \"kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.577283 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.577263    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.577468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.577448    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.578128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.578103    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.580100 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.580077    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.668842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.664731    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v67m\" (UniqueName: \"kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m\") pod \"maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:37.711111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:37.709054    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:38.013237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:38.013202    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48b781b-956d-4513-9234-820c801a62c4" path="/var/lib/kubelet/pods/b48b781b-956d-4513-9234-820c801a62c4/volumes"
Oct 30 20:11:38.169705 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:11:38.169667    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853a348f_d806_4eb0_aa18_71fc6bf14eed.slice/crio-348f74010b60841ba3309dbb664e1c8728195c144efab1d7a37b49ee00d0ff6d WatchSource:0}: Error finding container 348f74010b60841ba3309dbb664e1c8728195c144efab1d7a37b49ee00d0ff6d: Status 404 returned error can't find the container with id 348f74010b60841ba3309dbb664e1c8728195c144efab1d7a37b49ee00d0ff6d
Oct 30 20:11:38.170711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:38.170686    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"]
Oct 30 20:11:38.472616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:38.472577    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z" event={"ID":"853a348f-d806-4eb0-aa18-71fc6bf14eed","Type":"ContainerStarted","Data":"f9833c61cbcbf2b37a1e436511499f5bc05138a87286831d3f4d81775ac40459"}
Oct 30 20:11:38.472834 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:38.472641    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z" event={"ID":"853a348f-d806-4eb0-aa18-71fc6bf14eed","Type":"ContainerStarted","Data":"348f74010b60841ba3309dbb664e1c8728195c144efab1d7a37b49ee00d0ff6d"}
Oct 30 20:11:38.594504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:38.594430    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z" podStartSLOduration=1.5944124290000001 podStartE2EDuration="1.594412429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:38.593008413 +0000 UTC m=+1794.757410057" watchObservedRunningTime="2026-10-30 20:11:38.594412429 +0000 UTC m=+1794.758814072"
Oct 30 20:11:39.487073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:39.486992    4917 generic.go:356] "Generic (PLEG): container finished" podID="853a348f-d806-4eb0-aa18-71fc6bf14eed" containerID="f9833c61cbcbf2b37a1e436511499f5bc05138a87286831d3f4d81775ac40459" exitCode=0
Oct 30 20:11:39.487436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:39.487117    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z" event={"ID":"853a348f-d806-4eb0-aa18-71fc6bf14eed","Type":"ContainerDied","Data":"f9833c61cbcbf2b37a1e436511499f5bc05138a87286831d3f4d81775ac40459"}
Oct 30 20:11:39.681082 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:11:39.681045    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:11:40.672756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.672729    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:40.730591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.730560    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca\") pod \"853a348f-d806-4eb0-aa18-71fc6bf14eed\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") "
Oct 30 20:11:40.730756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.730737    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m\" (UniqueName: \"kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m\") pod \"853a348f-d806-4eb0-aa18-71fc6bf14eed\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") "
Oct 30 20:11:40.730852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.730818    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca\") pod \"853a348f-d806-4eb0-aa18-71fc6bf14eed\" (UID: \"853a348f-d806-4eb0-aa18-71fc6bf14eed\") "
Oct 30 20:11:40.731165 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.731139    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca" pod "853a348f-d806-4eb0-aa18-71fc6bf14eed" (UID: "853a348f-d806-4eb0-aa18-71fc6bf14eed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:40.731509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.731491    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.732705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.732684    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca" pod "853a348f-d806-4eb0-aa18-71fc6bf14eed" (UID: "853a348f-d806-4eb0-aa18-71fc6bf14eed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:40.732878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.732848    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m" pod "853a348f-d806-4eb0-aa18-71fc6bf14eed" (UID: "853a348f-d806-4eb0-aa18-71fc6bf14eed"). InnerVolumeSpecName "kube-api-access-4v67m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:40.832885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.832800    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853a348f-d806-4eb0-aa18-71fc6bf14eed-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.832885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:40.832825    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v67m\" (UniqueName: \"kubernetes.io/projected/853a348f-d806-4eb0-aa18-71fc6bf14eed-kube-api-access-4v67m\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.505653 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:41.505619    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"
Oct 30 20:11:41.505869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:41.505662    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z" event={"ID":"853a348f-d806-4eb0-aa18-71fc6bf14eed","Type":"ContainerDied","Data":"348f74010b60841ba3309dbb664e1c8728195c144efab1d7a37b49ee00d0ff6d"}
Oct 30 20:11:41.505869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:41.505710    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="348f74010b60841ba3309dbb664e1c8728195c144efab1d7a37b49ee00d0ff6d"
Oct 30 20:11:42.080677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:42.080609    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"]
Oct 30 20:11:42.089140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:42.089110    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ea0add-9rk45"]
Oct 30 20:11:44.007497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:44.007466    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825f4298-0eb6-4193-8a15-5d5a11703b0a" path="/var/lib/kubelet/pods/825f4298-0eb6-4193-8a15-5d5a11703b0a/volumes"
Oct 30 20:11:44.236386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:44.236360    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:11:44.280508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:44.280429    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:11:44.613166 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:44.613087    4917 scope.go:122] "RemoveContainer" containerID="b20476486ba9136c3273a31dfc8a4bfea2cdd1f9f14632a73dd3a7f3d8201e88"
Oct 30 20:11:49.219265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:49.219230    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"]
Oct 30 20:11:54.220040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.219958    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx" podUID="b3a2ac90-0da7-48a0-bfce-49d42493ea94" containerName="payload-pre-processing" containerID="cri-o://4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0" gracePeriod=25
Oct 30 20:11:54.526897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.526869    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:11:54.584706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.584672    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume\") pod \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") "
Oct 30 20:11:54.584706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.584716    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5\" (UniqueName: \"kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5\") pod \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\" (UID: \"b3a2ac90-0da7-48a0-bfce-49d42493ea94\") "
Oct 30 20:11:54.585227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.585198    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume" pod "b3a2ac90-0da7-48a0-bfce-49d42493ea94" (UID: "b3a2ac90-0da7-48a0-bfce-49d42493ea94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:54.585485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.585460    4917 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a2ac90-0da7-48a0-bfce-49d42493ea94-plugins-config-volume\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:54.587334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.587304    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5" pod "b3a2ac90-0da7-48a0-bfce-49d42493ea94" (UID: "b3a2ac90-0da7-48a0-bfce-49d42493ea94"). InnerVolumeSpecName "kube-api-access-2m9v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:54.630965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.630931    4917 generic.go:356] "Generic (PLEG): container finished" podID="b3a2ac90-0da7-48a0-bfce-49d42493ea94" containerID="4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0" exitCode=0
Oct 30 20:11:54.631092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.630975    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"
Oct 30 20:11:54.631092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.631002    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx" event={"ID":"b3a2ac90-0da7-48a0-bfce-49d42493ea94","Type":"ContainerDied","Data":"4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0"}
Oct 30 20:11:54.631092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.631062    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx" event={"ID":"b3a2ac90-0da7-48a0-bfce-49d42493ea94","Type":"ContainerDied","Data":"35eea8f6c8cd13537c2a0e2b894b7d18dca40c703b76ceb7c03ec29042bfd57a"}
Oct 30 20:11:54.631092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.631084    4917 scope.go:122] "RemoveContainer" containerID="4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0"
Oct 30 20:11:54.642566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.642549    4917 scope.go:122] "RemoveContainer" containerID="4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0"
Oct 30 20:11:54.642904 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:11:54.642882    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0\": container with ID starting with 4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0 not found: ID does not exist" containerID="4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0"
Oct 30 20:11:54.643022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.642910    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0"} err="failed to get container status \"4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0\": rpc error: code = NotFound desc = could not find container \"4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0\": container with ID starting with 4c3f12aa21ac32ed55026d9b7b759fd86fa5b18ada1de141fa4a2c719dd83bb0 not found: ID does not exist"
Oct 30 20:11:54.691839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.691792    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9v5\" (UniqueName: \"kubernetes.io/projected/b3a2ac90-0da7-48a0-bfce-49d42493ea94-kube-api-access-2m9v5\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:54.775556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.775397    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"]
Oct 30 20:11:54.796533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:54.796478    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7ca5342-6d5655cc9f-kn9gx"]
Oct 30 20:11:56.004074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:56.004042    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a2ac90-0da7-48a0-bfce-49d42493ea94" path="/var/lib/kubelet/pods/b3a2ac90-0da7-48a0-bfce-49d42493ea94/volumes"
Oct 30 20:11:59.056036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:59.056005    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"]
Oct 30 20:11:59.091848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:11:59.091796    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d558de14-d555l"]
Oct 30 20:12:00.004650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:00.004608    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81cb4e0-c353-4938-82c2-b5953f1a49f2" path="/var/lib/kubelet/pods/c81cb4e0-c353-4938-82c2-b5953f1a49f2/volumes"
Oct 30 20:12:06.057743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:06.057707    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"]
Oct 30 20:12:06.105432 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:06.105400    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2771248-7xwnc"]
Oct 30 20:12:08.007266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:08.007233    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed9ecde-4773-47d7-b291-2376a50d1063" path="/var/lib/kubelet/pods/1ed9ecde-4773-47d7-b291-2376a50d1063/volumes"
Oct 30 20:12:09.683917 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:12:09.683871    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:12:09.809404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.809366    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"]
Oct 30 20:12:09.810119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.810097    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b3a2ac90-0da7-48a0-bfce-49d42493ea94" containerName="payload-pre-processing"
Oct 30 20:12:09.810119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.810119    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a2ac90-0da7-48a0-bfce-49d42493ea94" containerName="payload-pre-processing"
Oct 30 20:12:09.810306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.810192    4917 cpu_manager.go:414] "containerMap: removing container" podUID="853a348f-d806-4eb0-aa18-71fc6bf14eed" containerName="revoke-keys"
Oct 30 20:12:09.810306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.810202    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853a348f-d806-4eb0-aa18-71fc6bf14eed" containerName="revoke-keys"
Oct 30 20:12:09.810408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.810383    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="853a348f-d806-4eb0-aa18-71fc6bf14eed" containerName="revoke-keys"
Oct 30 20:12:09.810408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.810404    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a2ac90-0da7-48a0-bfce-49d42493ea94" containerName="payload-pre-processing"
Oct 30 20:12:09.814328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.814308    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912408    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912637 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912503    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912637 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912563    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912643    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912677    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912816    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.912900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912848    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.913007 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.912906    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.913199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.913172    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4g5l\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:09.918271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:09.918246    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"]
Oct 30 20:12:10.014736 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.014651    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.014736 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.014731    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.014828    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.014990    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015028    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015102    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015126    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015157    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015209    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015401 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015291    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4g5l\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015479    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015562    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015617    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.015842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.015818    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.017167 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.017145    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.017540 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.017520    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.077093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.077054    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.077286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.077257    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4g5l\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bmtzs\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.128554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.128523    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:10.566142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.566105    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"]
Oct 30 20:12:10.567869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.567833    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:12:10.567993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.567896    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:12:10.567993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.567929    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:12:10.784810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.784742    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" event={"ID":"8fa81269-e74c-4900-b895-33b56428061d","Type":"ContainerStarted","Data":"98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403"}
Oct 30 20:12:10.785204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.784820    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" event={"ID":"8fa81269-e74c-4900-b895-33b56428061d","Type":"ContainerStarted","Data":"303b35a8f94aa9729dececb34fbf13412754bb21e2d64d514ca5c6eaaa708db0"}
Oct 30 20:12:10.909489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:10.909383    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podStartSLOduration=1.9093690190000001 podStartE2EDuration="1.909369019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:10.908028029 +0000 UTC m=+1827.072429694" watchObservedRunningTime="2026-10-30 20:12:10.909369019 +0000 UTC m=+1827.073770732"
Oct 30 20:12:11.132564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:11.131096    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:11.135594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:11.133706    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:11.135594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:11.133831    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:12.129261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:12.129224    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:12.129684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:12.129288    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:13.129836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:13.129793    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:13.130303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:13.129854    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:14.129405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:14.129370    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:14.129592 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:14.129426    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:15.287794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:15.283981    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:15.287794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:15.284605    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:15.345808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:15.338852    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"]
Oct 30 20:12:15.598574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:15.598484    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4a87a0-hq67l"]
Oct 30 20:12:16.009545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:16.009509    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e312f4f5-febd-4877-b571-8e4c8000699d" path="/var/lib/kubelet/pods/e312f4f5-febd-4877-b571-8e4c8000699d/volumes"
Oct 30 20:12:16.129856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:16.129816    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:16.130023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:16.129895    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:17.130109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:17.130072    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:17.130730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:17.130694    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:18.127901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:18.127867    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"]
Oct 30 20:12:18.138125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:18.138092    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:12:18.139363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:18.139320    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:12:18.154894 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:18.141621    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-27915880-s9v7n"]
Oct 30 20:12:19.707748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:19.707712    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:19.708909 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:19.708881    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:19.710124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:19.710088    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:12:19.961422 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:19.961324    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"]
Oct 30 20:12:19.967831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:19.967789    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80e4d90e-96qc8"]
Oct 30 20:12:20.021193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:20.020556    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b947c0-480d-43b5-87f3-daa484266f27" path="/var/lib/kubelet/pods/24b947c0-480d-43b5-87f3-daa484266f27/volumes"
Oct 30 20:12:20.022236 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:20.022198    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa255090-f05a-4f09-9d3e-81e73a82d827" path="/var/lib/kubelet/pods/aa255090-f05a-4f09-9d3e-81e73a82d827/volumes"
Oct 30 20:12:24.059661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:24.059587    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"]
Oct 30 20:12:24.109778 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:24.109732    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be70943b-pwbhp"]
Oct 30 20:12:26.006927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:26.006890    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e35c54c-1810-4fcf-ab42-28201a747b07" path="/var/lib/kubelet/pods/4e35c54c-1810-4fcf-ab42-28201a747b07/volumes"
Oct 30 20:12:27.970511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:27.970473    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"]
Oct 30 20:12:27.980296 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:27.980260    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7ca5342-sjc7z"]
Oct 30 20:12:28.015747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.015708    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853a348f-d806-4eb0-aa18-71fc6bf14eed" path="/var/lib/kubelet/pods/853a348f-d806-4eb0-aa18-71fc6bf14eed/volumes"
Oct 30 20:12:28.701546 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.701510    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"]
Oct 30 20:12:28.707284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.707259    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.764209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.764172    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"]
Oct 30 20:12:28.807285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.807243    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:28.807509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.807294    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:28.807597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.807551    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5z7j\" (UniqueName: \"kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.807843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.807817    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.807971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.807953    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.811556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.811541    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:12:28.811603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.811542    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:28.909493 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.909460    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.909647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.909610    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5z7j\" (UniqueName: \"kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.909713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.909707    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.910429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.910406    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.912071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.912051    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:28.972588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:28.972515    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5z7j\" (UniqueName: \"kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j\") pod \"maas-api-revoke-sub-3f8a1a65-fgghk\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:29.023053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:29.023018    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:29.416009 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:12:29.415970    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3fd2c0_80e6_4000_9f87_d7698cac02c3.slice/crio-592af0d489bf06cf9293dcbbc4d55236623f77c14f85d2fd364e1d8975bc283a WatchSource:0}: Error finding container 592af0d489bf06cf9293dcbbc4d55236623f77c14f85d2fd364e1d8975bc283a: Status 404 returned error can't find the container with id 592af0d489bf06cf9293dcbbc4d55236623f77c14f85d2fd364e1d8975bc283a
Oct 30 20:12:29.417801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:29.417756    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"]
Oct 30 20:12:30.014225 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:30.014143    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk" event={"ID":"0a3fd2c0-80e6-4000-9f87-d7698cac02c3","Type":"ContainerStarted","Data":"4f68ca8563a8cc456bebb246fedc0b358e0c041e2820727c81fff9b145ae5fe0"}
Oct 30 20:12:30.014225 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:30.014198    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk" event={"ID":"0a3fd2c0-80e6-4000-9f87-d7698cac02c3","Type":"ContainerStarted","Data":"592af0d489bf06cf9293dcbbc4d55236623f77c14f85d2fd364e1d8975bc283a"}
Oct 30 20:12:30.136986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:30.136929    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk" podStartSLOduration=2.136912573 podStartE2EDuration="2.136912573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:30.131653059 +0000 UTC m=+1846.296054702" watchObservedRunningTime="2026-10-30 20:12:30.136912573 +0000 UTC m=+1846.301314214"
Oct 30 20:12:31.030196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:31.030162    4917 generic.go:356] "Generic (PLEG): container finished" podID="0a3fd2c0-80e6-4000-9f87-d7698cac02c3" containerID="4f68ca8563a8cc456bebb246fedc0b358e0c041e2820727c81fff9b145ae5fe0" exitCode=0
Oct 30 20:12:31.030629 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:31.030253    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk" event={"ID":"0a3fd2c0-80e6-4000-9f87-d7698cac02c3","Type":"ContainerDied","Data":"4f68ca8563a8cc456bebb246fedc0b358e0c041e2820727c81fff9b145ae5fe0"}
Oct 30 20:12:32.484512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.484437    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:32.507860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.507830    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca\") pod \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") "
Oct 30 20:12:32.508011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.507871    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j\" (UniqueName: \"kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j\") pod \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") "
Oct 30 20:12:32.508083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.508067    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca\") pod \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\" (UID: \"0a3fd2c0-80e6-4000-9f87-d7698cac02c3\") "
Oct 30 20:12:32.508452 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.508423    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca" pod "0a3fd2c0-80e6-4000-9f87-d7698cac02c3" (UID: "0a3fd2c0-80e6-4000-9f87-d7698cac02c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:32.508639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.508622    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:32.510662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.510615    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca" pod "0a3fd2c0-80e6-4000-9f87-d7698cac02c3" (UID: "0a3fd2c0-80e6-4000-9f87-d7698cac02c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:32.510841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.510741    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j" pod "0a3fd2c0-80e6-4000-9f87-d7698cac02c3" (UID: "0a3fd2c0-80e6-4000-9f87-d7698cac02c3"). InnerVolumeSpecName "kube-api-access-r5z7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:32.610221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.610182    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:32.610221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:32.610219    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5z7j\" (UniqueName: \"kubernetes.io/projected/0a3fd2c0-80e6-4000-9f87-d7698cac02c3-kube-api-access-r5z7j\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:33.231504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:33.231472    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"
Oct 30 20:12:33.231504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:33.231493    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk" event={"ID":"0a3fd2c0-80e6-4000-9f87-d7698cac02c3","Type":"ContainerDied","Data":"592af0d489bf06cf9293dcbbc4d55236623f77c14f85d2fd364e1d8975bc283a"}
Oct 30 20:12:33.231735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:33.231548    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592af0d489bf06cf9293dcbbc4d55236623f77c14f85d2fd364e1d8975bc283a"
Oct 30 20:12:34.043397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:34.043361    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:12:34.057325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:34.055485    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"]
Oct 30 20:12:34.062227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:34.062200    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7be398-76rb7"]
Oct 30 20:12:36.003958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:36.003928    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfba6e83-3211-4870-b9f4-74aebac007a2" path="/var/lib/kubelet/pods/dfba6e83-3211-4870-b9f4-74aebac007a2/volumes"
Oct 30 20:12:39.699217 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:12:39.699176    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:12:44.060531 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.060495    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"]
Oct 30 20:12:44.104648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.104616    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5485ca1b-zfnvc"]
Oct 30 20:12:44.646186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.646154    4917 scope.go:122] "RemoveContainer" containerID="3508104b270cdab46c31bc097ad6928d19ff8e5b73044930de0a28f12f95b53e"
Oct 30 20:12:44.656652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.656628    4917 scope.go:122] "RemoveContainer" containerID="a704bb8007d803412c5d5814fcead2836e7b09d5b46942f0ed86e7cdf4a9fac4"
Oct 30 20:12:44.666333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.666314    4917 scope.go:122] "RemoveContainer" containerID="ba034cfad8fa022fa6dae9cac4d244cf04e8a9b9539af3435e01422e59735f6b"
Oct 30 20:12:44.675985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.675964    4917 scope.go:122] "RemoveContainer" containerID="7c37f0d5aaede54b73f3fc1d7b1963ac9ad47d59f98e9fbc31ca678a17fb14fd"
Oct 30 20:12:44.685154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:44.685130    4917 scope.go:122] "RemoveContainer" containerID="209bc1ff114be921e9fff8e2383fc730105ba361a25bf47f22aedc5a1b09b4b9"
Oct 30 20:12:46.005263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:46.005230    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d12d49f-ed02-4b5e-9045-abb8841d2c78" path="/var/lib/kubelet/pods/6d12d49f-ed02-4b5e-9045-abb8841d2c78/volumes"
Oct 30 20:12:51.109062 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:51.109018    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"]
Oct 30 20:12:51.114987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:51.114947    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a1bb42-8jf9f"]
Oct 30 20:12:52.007834 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:12:52.007796    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70646b36-d40d-4fbf-b398-9d4a3378fe3e" path="/var/lib/kubelet/pods/70646b36-d40d-4fbf-b398-9d4a3378fe3e/volumes"
Oct 30 20:13:01.057093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:01.057056    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"]
Oct 30 20:13:01.107136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:01.107103    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2048026-46ssp"]
Oct 30 20:13:02.004722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:02.004684    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9abb9c-4295-4440-afa6-ec70680f6e65" path="/var/lib/kubelet/pods/df9abb9c-4295-4440-afa6-ec70680f6e65/volumes"
Oct 30 20:13:07.057420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:07.057386    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"]
Oct 30 20:13:07.108628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:07.108590    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8752d712-8btqj"]
Oct 30 20:13:08.014607 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:08.014567    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9" path="/var/lib/kubelet/pods/3aa12c27-3ab6-4fc5-911d-3b1a82a2f7e9/volumes"
Oct 30 20:13:09.680555 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:13:09.680507    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:13:10.819380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:10.819336    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:13:12.077564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:12.077521    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"]
Oct 30 20:13:12.126611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:12.126564    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23660b5-lsv5q"]
Oct 30 20:13:14.005481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:14.005448    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd9f7b5-311e-4d99-b6db-f93d071fec2d" path="/var/lib/kubelet/pods/afd9f7b5-311e-4d99-b6db-f93d071fec2d/volumes"
Oct 30 20:13:19.054436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:19.054394    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"]
Oct 30 20:13:19.104198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:19.104162    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12624a1-q8xmf"]
Oct 30 20:13:20.005098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:20.005063    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4bea89-c2b7-482f-a4d5-3c7b6b75de4e" path="/var/lib/kubelet/pods/db4bea89-c2b7-482f-a4d5-3c7b6b75de4e/volumes"
Oct 30 20:13:22.461156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.461116    4917 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:13:22.461676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.461192    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:13:22.724720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.724638    4917 generic.go:356] "Generic (PLEG): container finished" podID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerID="1365b76ba70512b221b6bed52067904bb3e199c3f0cd90cf026016a4bf25a726" exitCode=0
Oct 30 20:13:22.724902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.724712    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" event={"ID":"e280fd88-2d98-449d-ab3d-55ad7bf216f8","Type":"ContainerDied","Data":"1365b76ba70512b221b6bed52067904bb3e199c3f0cd90cf026016a4bf25a726"}
Oct 30 20:13:22.725203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.725184    4917 scope.go:122] "RemoveContainer" containerID="1365b76ba70512b221b6bed52067904bb3e199c3f0cd90cf026016a4bf25a726"
Oct 30 20:13:22.726805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.726757    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:13:22.726885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.726847    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:13:22.726885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:22.726880    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:13:23.734804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:23.734739    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" event={"ID":"e280fd88-2d98-449d-ab3d-55ad7bf216f8","Type":"ContainerStarted","Data":"f78892d065ef1e95a8b66d41b817ec89be99a0f6d4c8a42b8c18fff516c99212"}
Oct 30 20:13:24.727895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:24.727859    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:13:24.728076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:24.728025    4917 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:13:24.728120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:24.728075    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:13:25.728558 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:25.728528    4917 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:13:25.729083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:25.728643    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:13:26.728244 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:26.728207    4917 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:13:26.728450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:26.728265    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:13:27.728482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:27.728444    4917 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:13:27.728945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:27.728516    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:13:28.728023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:28.727988    4917 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:13:28.728202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:28.728039    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 20:13:29.732486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:29.732453    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:13:29.732988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:29.732975    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:13:29.733754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:29.733732    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:13:39.705097 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:13:39.705020    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:13:44.741899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:44.741867    4917 scope.go:122] "RemoveContainer" containerID="b85f5d3a8f3aa5f662611adabef31db650c7d00df8ea6eff571bddaf29e55853"
Oct 30 20:13:44.752655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:44.752636    4917 scope.go:122] "RemoveContainer" containerID="b664dbab43abf58fe0186abf36bc973fb2b6753c0d9e6ae455fbe666e5b212df"
Oct 30 20:13:44.762686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:44.762668    4917 scope.go:122] "RemoveContainer" containerID="7bca86fa8f8da9b17c2b08927525ba70de6f76fb1cb5f2c82bb235127fa42436"
Oct 30 20:13:44.772590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:44.772562    4917 scope.go:122] "RemoveContainer" containerID="bca897581be9b0b246faf25cc11cd092defd0d59272b12946f6bb19248e02966"
Oct 30 20:13:44.782874 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:44.782854    4917 scope.go:122] "RemoveContainer" containerID="a6f11a06c96c3680e1578859cd30f4a10b21e232ee8bb2bc19668a66bbf5b883"
Oct 30 20:13:44.792625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:44.792597    4917 scope.go:122] "RemoveContainer" containerID="b36532585f2b425b6efe8cf77d1eaea59edcaf92ff7ee40594df0d4c5c8b88d8"
Oct 30 20:13:46.056984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:46.056949    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"]
Oct 30 20:13:46.107106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:46.107061    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd16f2ae-7gdrp"]
Oct 30 20:13:48.010416 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:48.010384    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3893bc5-d086-41bf-9139-2f541f66e76a" path="/var/lib/kubelet/pods/b3893bc5-d086-41bf-9139-2f541f66e76a/volumes"
Oct 30 20:13:48.059318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:48.059289    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"]
Oct 30 20:13:48.109329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:48.109295    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b8bc59-pv2hh"]
Oct 30 20:13:50.004487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:50.004451    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2204ec6-3fec-44da-9db9-02f1e06d0a83" path="/var/lib/kubelet/pods/c2204ec6-3fec-44da-9db9-02f1e06d0a83/volumes"
Oct 30 20:13:58.122891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:58.122797    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"]
Oct 30 20:13:58.167037 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:58.167002    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef8074a-59lcx"]
Oct 30 20:13:59.068543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:59.068506    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"]
Oct 30 20:13:59.118247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:13:59.118212    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-083d6d38-t45mx"]
Oct 30 20:14:00.003534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:00.003497    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1455b048-4f80-431a-ae23-8ecd28bf9478" path="/var/lib/kubelet/pods/1455b048-4f80-431a-ae23-8ecd28bf9478/volumes"
Oct 30 20:14:00.004075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:00.004059    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f964f34-2bdb-46ce-aa0c-1ca959d9fc52" path="/var/lib/kubelet/pods/7f964f34-2bdb-46ce-aa0c-1ca959d9fc52/volumes"
Oct 30 20:14:04.772602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:04.772565    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"]
Oct 30 20:14:04.773052 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:04.772813    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" containerID="cri-o://89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6" gracePeriod=30
Oct 30 20:14:04.831739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:04.831700    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"]
Oct 30 20:14:04.832020 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:04.831975    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy" containerID="cri-o://542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708" gracePeriod=30
Oct 30 20:14:08.707297 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:08.707242    4917 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:14:08.707808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:08.707334    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:14:09.057101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:09.057024    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"]
Oct 30 20:14:09.107468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:09.107429    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b298a625-4tbjf"]
Oct 30 20:14:09.680470 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:14:09.680432    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:14:10.019928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.018275    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5354a6-751b-4959-9982-f061b80cf271" path="/var/lib/kubelet/pods/0d5354a6-751b-4959-9982-f061b80cf271/volumes"
Oct 30 20:14:10.129828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.129801    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:14:10.174462 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.174431    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:14:10.178179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.178154    4917 generic.go:356] "Generic (PLEG): container finished" podID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerID="542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708" exitCode=0
Oct 30 20:14:10.178286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.178196    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"
Oct 30 20:14:10.178286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.178232    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" event={"ID":"608c759b-1b44-444e-a4b1-6b1482b59f59","Type":"ContainerDied","Data":"542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708"}
Oct 30 20:14:10.178400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.178284    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592" event={"ID":"608c759b-1b44-444e-a4b1-6b1482b59f59","Type":"ContainerDied","Data":"56cd7f9beca70a5a002a8415ebbffa338ac38ef4120b82b8421ee1767f212c5f"}
Oct 30 20:14:10.178400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.178301    4917 scope.go:122] "RemoveContainer" containerID="542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708"
Oct 30 20:14:10.180099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.180075    4917 generic.go:356] "Generic (PLEG): container finished" podID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerID="89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6" exitCode=0
Oct 30 20:14:10.180183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.180110    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"
Oct 30 20:14:10.180183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.180158    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" event={"ID":"4028bea6-3a70-4980-890f-4a86c7d44b2e","Type":"ContainerDied","Data":"89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6"}
Oct 30 20:14:10.180278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.180210    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr" event={"ID":"4028bea6-3a70-4980-890f-4a86c7d44b2e","Type":"ContainerDied","Data":"2c1e8902e32565ce25a3827b48521ff28eda248edc352fbded688f9c57246f27"}
Oct 30 20:14:10.192060 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.192033    4917 scope.go:122] "RemoveContainer" containerID="542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708"
Oct 30 20:14:10.194416 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:14:10.194391    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708\": container with ID starting with 542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708 not found: ID does not exist" containerID="542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708"
Oct 30 20:14:10.194581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.194558    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708"} err="failed to get container status \"542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708\": rpc error: code = NotFound desc = could not find container \"542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708\": container with ID starting with 542c7dff31fdea8df0de315e6cb48b9f5ffa80d1ca70fa81185af21be8944708 not found: ID does not exist"
Oct 30 20:14:10.194705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.194682    4917 scope.go:122] "RemoveContainer" containerID="89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6"
Oct 30 20:14:10.205345 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.205331    4917 scope.go:122] "RemoveContainer" containerID="89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6"
Oct 30 20:14:10.205602 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:14:10.205581    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6\": container with ID starting with 89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6 not found: ID does not exist" containerID="89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6"
Oct 30 20:14:10.205647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.205607    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6"} err="failed to get container status \"89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6\": rpc error: code = NotFound desc = could not find container \"89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6\": container with ID starting with 89e1db88b86b82524cb5e804dca00e58f4137450630b6904f41e42faf37a76f6 not found: ID does not exist"
Oct 30 20:14:10.278808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.278743    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.278991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.278814    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.278991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.278849    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.278991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.278953    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279182 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279021    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279182 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279047    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279182 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279104    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279182 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279110    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.279392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279197    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279254    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279342    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279468    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279472    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.279568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279516    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279549    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279676    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279717    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.279922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279804    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279839    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy\") pod \"4028bea6-3a70-4980-890f-4a86c7d44b2e\" (UID: \"4028bea6-3a70-4980-890f-4a86c7d44b2e\") "
Oct 30 20:14:10.279922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.279899    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert\") pod \"608c759b-1b44-444e-a4b1-6b1482b59f59\" (UID: \"608c759b-1b44-444e-a4b1-6b1482b59f59\") "
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.281419    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.281466    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.282230    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.283444    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.283492    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.283802    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.285058    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.285173    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.285191    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.285206    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.285220    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:10.290809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.288855    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "kube-api-access-826xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:10.292139 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.292111    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.293374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.293072    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "kube-api-access-6gqxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:10.294902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.294873    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:10.300035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.300011    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:10.300393 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.300183    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo" pod "4028bea6-3a70-4980-890f-4a86c7d44b2e" (UID: "4028bea6-3a70-4980-890f-4a86c7d44b2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:10.300818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.300798    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy" pod "608c759b-1b44-444e-a4b1-6b1482b59f59" (UID: "608c759b-1b44-444e-a4b1-6b1482b59f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:10.383374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383342    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383369    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383381    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383389    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383398    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608c759b-1b44-444e-a4b1-6b1482b59f59-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383407    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-826xx\" (UniqueName: \"kubernetes.io/projected/608c759b-1b44-444e-a4b1-6b1482b59f59-kube-api-access-826xx\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383417    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqxp\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-kube-api-access-6gqxp\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383425    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383433    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383442    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383450    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383458    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bea6-3a70-4980-890f-4a86c7d44b2e-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383466    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608c759b-1b44-444e-a4b1-6b1482b59f59-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383474    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383482    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608c759b-1b44-444e-a4b1-6b1482b59f59-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.383612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.383490    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bea6-3a70-4980-890f-4a86c7d44b2e-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.616991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.616953    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"]
Oct 30 20:14:10.621289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.621255    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-88592"]
Oct 30 20:14:10.728706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.728665    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"]
Oct 30 20:14:10.732002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:10.731973    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-kthkr"]
Oct 30 20:14:11.904570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.904535    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"]
Oct 30 20:14:11.905231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905214    4917 cpu_manager.go:414] "containerMap: removing container" podUID="0a3fd2c0-80e6-4000-9f87-d7698cac02c3" containerName="revoke-keys"
Oct 30 20:14:11.905280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905233    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3fd2c0-80e6-4000-9f87-d7698cac02c3" containerName="revoke-keys"
Oct 30 20:14:11.905280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905247    4917 cpu_manager.go:414] "containerMap: removing container" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy"
Oct 30 20:14:11.905280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905252    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy"
Oct 30 20:14:11.905280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905279    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy"
Oct 30 20:14:11.905420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905285    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy"
Oct 30 20:14:11.905420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905401    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3fd2c0-80e6-4000-9f87-d7698cac02c3" containerName="revoke-keys"
Oct 30 20:14:11.905420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905413    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" containerName="istio-proxy"
Oct 30 20:14:11.905510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.905426    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" containerName="istio-proxy"
Oct 30 20:14:11.911075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.911047    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:11.964606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:11.964577    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-77fcfa-openshift-default-dockercfg-d7dp2\""
Oct 30 20:14:12.007509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.007477    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4028bea6-3a70-4980-890f-4a86c7d44b2e" path="/var/lib/kubelet/pods/4028bea6-3a70-4980-890f-4a86c7d44b2e/volumes"
Oct 30 20:14:12.008307 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.008278    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.008471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.008429    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608c759b-1b44-444e-a4b1-6b1482b59f59" path="/var/lib/kubelet/pods/608c759b-1b44-444e-a4b1-6b1482b59f59/volumes"
Oct 30 20:14:12.008582 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.008504    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.008652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.008617    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.008717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.008650    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.009319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.009292    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.009444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.009360    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.009444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.009394    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.009566 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.009506    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.009624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.009585    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgppq\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.035208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.035096    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"]
Oct 30 20:14:12.132405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132361    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132453    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132486    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132608    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132676    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgppq\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132799    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132873 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132830    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132878    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.132935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.132908    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.133408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.133326    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.134128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.133996    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.134319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.134288    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.134403 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.134364    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.140240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.140210    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.151497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.151464    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.155524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.155457    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.196527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.196494    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.199155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.199119    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgppq\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq\") pod \"e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") " pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.226009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.225977    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:12.703707 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:14:12.703670    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795942c1_11a0_4218_a931_a6a0807d8adf.slice/crio-0b0e1160ec6c02a70c37760380c05d17d7697868d02563f441e7f87526656711 WatchSource:0}: Error finding container 0b0e1160ec6c02a70c37760380c05d17d7697868d02563f441e7f87526656711: Status 404 returned error can't find the container with id 0b0e1160ec6c02a70c37760380c05d17d7697868d02563f441e7f87526656711
Oct 30 20:14:12.706382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.706354    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"]
Oct 30 20:14:12.706808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.706759    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:14:12.706859 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.706844    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:14:12.706893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:12.706871    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:14:13.211517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:13.211476    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" event={"ID":"795942c1-11a0-4218-a931-a6a0807d8adf","Type":"ContainerStarted","Data":"6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"}
Oct 30 20:14:13.211967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:13.211537    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" event={"ID":"795942c1-11a0-4218-a931-a6a0807d8adf","Type":"ContainerStarted","Data":"0b0e1160ec6c02a70c37760380c05d17d7697868d02563f441e7f87526656711"}
Oct 30 20:14:13.226437 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:13.226408    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:13.232447 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:13.232422    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:13.338394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:13.338341    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podStartSLOduration=2.338327881 podStartE2EDuration="2.338327881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:13.334233578 +0000 UTC m=+1949.498635221" watchObservedRunningTime="2026-10-30 20:14:13.338327881 +0000 UTC m=+1949.502729524"
Oct 30 20:14:14.220895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:14.220862    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:14.221913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:14.221888    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:14:31.056200 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:31.056163    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"]
Oct 30 20:14:31.104034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:31.104003    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d401f-rgws2"]
Oct 30 20:14:32.004194 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:32.004163    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3954af2c-3296-4e4b-b1d6-b8bd75ae3073" path="/var/lib/kubelet/pods/3954af2c-3296-4e4b-b1d6-b8bd75ae3073/volumes"
Oct 30 20:14:33.145413 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:33.145378    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:14:35.864842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:35.864805    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"]
Oct 30 20:14:35.865288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:35.865031    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy" containerID="cri-o://3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce" gracePeriod=30
Oct 30 20:14:35.918699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:35.918665    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"]
Oct 30 20:14:35.918956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:35.918909    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy" containerID="cri-o://dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc" gracePeriod=30
Oct 30 20:14:38.066532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:38.066489    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"]
Oct 30 20:14:38.117013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:38.116980    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-084205fb-zwh8z"]
Oct 30 20:14:39.680286 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:14:39.680237    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:14:40.004729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:40.004655    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd5a668-2bb3-420d-b512-9b549717b9a2" path="/var/lib/kubelet/pods/fbd5a668-2bb3-420d-b512-9b549717b9a2/volumes"
Oct 30 20:14:41.225600 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.225566    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:14:41.251329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.251306    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:14:41.393189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393159    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.393361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393216    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.393361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393260    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.393361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393288    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.393361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393326    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.393553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393379    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.393553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393411    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.393553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393434    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.393553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393518    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.393553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393551    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.393840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393557    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.393840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393571    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.393840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393601    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.393840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393625    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.393840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393694    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.393840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393761    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.394145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393901    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.394145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393959    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.394145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.393972    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:41.394145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394010    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.394145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394058    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs\") pod \"24b11eea-3dec-4bba-9b7c-140884a72905\" (UID: \"24b11eea-3dec-4bba-9b7c-140884a72905\") "
Oct 30 20:14:41.394145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394095    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs\") pod \"39b2dce0-514d-4ab3-8b24-5106f5613b97\" (UID: \"39b2dce0-514d-4ab3-8b24-5106f5613b97\") "
Oct 30 20:14:41.394584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394016    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.394584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394264    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:41.394584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394303    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.394584 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394513    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.394866 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394816    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.394866 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.394825    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.395220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395194    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b2dce0-514d-4ab3-8b24-5106f5613b97-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395223    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b11eea-3dec-4bba-9b7c-140884a72905-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395239    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395254    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395269    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395282    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395296    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395312    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395326    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.395349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.395340    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.396187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.396162    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "kube-api-access-kb4tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:41.396289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.396264    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:41.396643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.396616    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "kube-api-access-rsd7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:41.396965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.396944    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:41.396965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.396953    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.397082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.397073    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy" pod "39b2dce0-514d-4ab3-8b24-5106f5613b97" (UID: "39b2dce0-514d-4ab3-8b24-5106f5613b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:41.397645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.397623    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:41.397691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.397676    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token" pod "24b11eea-3dec-4bba-9b7c-140884a72905" (UID: "24b11eea-3dec-4bba-9b7c-140884a72905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:41.452931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.452845    4917 generic.go:356] "Generic (PLEG): container finished" podID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerID="dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc" exitCode=0
Oct 30 20:14:41.452931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.452883    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" event={"ID":"39b2dce0-514d-4ab3-8b24-5106f5613b97","Type":"ContainerDied","Data":"dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc"}
Oct 30 20:14:41.452931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.452915    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"
Oct 30 20:14:41.453212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.452950    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv" event={"ID":"39b2dce0-514d-4ab3-8b24-5106f5613b97","Type":"ContainerDied","Data":"d059c34b12078b83906f7cd1dee4e823449b32ebbd468ed86350c9f7bd0619e2"}
Oct 30 20:14:41.453212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.452983    4917 scope.go:122] "RemoveContainer" containerID="dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc"
Oct 30 20:14:41.454857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.454829    4917 generic.go:356] "Generic (PLEG): container finished" podID="24b11eea-3dec-4bba-9b7c-140884a72905" containerID="3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce" exitCode=0
Oct 30 20:14:41.454993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.454881    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"
Oct 30 20:14:41.455135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.455100    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" event={"ID":"24b11eea-3dec-4bba-9b7c-140884a72905","Type":"ContainerDied","Data":"3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce"}
Oct 30 20:14:41.455249 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.455180    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn" event={"ID":"24b11eea-3dec-4bba-9b7c-140884a72905","Type":"ContainerDied","Data":"6b4506024e7a1c58070e4fa21dca0784a156ef529f2c939a8400ce3759ac817a"}
Oct 30 20:14:41.467339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.467323    4917 scope.go:122] "RemoveContainer" containerID="dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc"
Oct 30 20:14:41.467638 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:14:41.467616    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc\": container with ID starting with dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc not found: ID does not exist" containerID="dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc"
Oct 30 20:14:41.467711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.467645    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc"} err="failed to get container status \"dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc\": rpc error: code = NotFound desc = could not find container \"dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc\": container with ID starting with dcbfec2ef0b42d4d3bdc75d2abc4c33565b33a942e24921cca144f78d80603cc not found: ID does not exist"
Oct 30 20:14:41.467711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.467671    4917 scope.go:122] "RemoveContainer" containerID="3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce"
Oct 30 20:14:41.478135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.478118    4917 scope.go:122] "RemoveContainer" containerID="3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce"
Oct 30 20:14:41.478378 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:14:41.478355    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce\": container with ID starting with 3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce not found: ID does not exist" containerID="3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce"
Oct 30 20:14:41.478440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.478389    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce"} err="failed to get container status \"3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce\": rpc error: code = NotFound desc = could not find container \"3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce\": container with ID starting with 3f610f2d9e96ee92b8ed407556f2f2a0d0da4d8d43cc9181e6b762021e8f4bce not found: ID does not exist"
Oct 30 20:14:41.496448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496422    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496445    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b11eea-3dec-4bba-9b7c-140884a72905-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496457    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496467    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4tj\" (UniqueName: \"kubernetes.io/projected/39b2dce0-514d-4ab3-8b24-5106f5613b97-kube-api-access-kb4tj\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496477    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496485    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsd7z\" (UniqueName: \"kubernetes.io/projected/24b11eea-3dec-4bba-9b7c-140884a72905-kube-api-access-rsd7z\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496503    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b11eea-3dec-4bba-9b7c-140884a72905-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.496571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.496513    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b2dce0-514d-4ab3-8b24-5106f5613b97-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:41.659018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.658978    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"]
Oct 30 20:14:41.663348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.663320    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-fq8cv"]
Oct 30 20:14:41.772877 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.772794    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"]
Oct 30 20:14:41.778416 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:41.778386    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-89bwn"]
Oct 30 20:14:42.008571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:42.008533    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" path="/var/lib/kubelet/pods/24b11eea-3dec-4bba-9b7c-140884a72905/volumes"
Oct 30 20:14:42.009133 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:42.009115    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" path="/var/lib/kubelet/pods/39b2dce0-514d-4ab3-8b24-5106f5613b97/volumes"
Oct 30 20:14:44.852110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:44.852082    4917 scope.go:122] "RemoveContainer" containerID="c12810d8fc97030dd127ad71e2aa56abadbc67f5c84bda125fa40d5ea18d983a"
Oct 30 20:14:44.862652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:44.862633    4917 scope.go:122] "RemoveContainer" containerID="b1206e40451bf2a8745e1e32ea6248488336c012e9c6207be1dd3df2ea73b9b6"
Oct 30 20:14:44.872778 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:44.872746    4917 scope.go:122] "RemoveContainer" containerID="d4e7bc07ff58e62bd757db327a2caa3e0e956fc3b982288b854e71ba243b05b5"
Oct 30 20:14:44.882247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:44.882223    4917 scope.go:122] "RemoveContainer" containerID="d868b6fde64b36d6ce156678d3b3cca323bc03b91b0645489eee817cbdbe77f0"
Oct 30 20:14:44.892395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:44.892373    4917 scope.go:122] "RemoveContainer" containerID="b41a35b656d444454f1323e81dee609eeddcbdec53564281d8f6150ba1f855f0"
Oct 30 20:14:44.901570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:44.901550    4917 scope.go:122] "RemoveContainer" containerID="892a619f665b2b657fbb037c9c293208961bc71eb05187b1f19077451b6db034"
Oct 30 20:14:54.619095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619057    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"]
Oct 30 20:14:54.619748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619731    4917 cpu_manager.go:414] "containerMap: removing container" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy"
Oct 30 20:14:54.619748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619748    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy"
Oct 30 20:14:54.619910 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619792    4917 cpu_manager.go:414] "containerMap: removing container" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy"
Oct 30 20:14:54.619910 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619798    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy"
Oct 30 20:14:54.619910 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619902    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b11eea-3dec-4bba-9b7c-140884a72905" containerName="istio-proxy"
Oct 30 20:14:54.620073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.619920    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b2dce0-514d-4ab3-8b24-5106f5613b97" containerName="istio-proxy"
Oct 30 20:14:54.623673 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.623655    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.677449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.677418    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5d2c70-openshift-default-dockercfg-2k8fx\""
Oct 30 20:14:54.698183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698143    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698197    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698261    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698488 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698388    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpx5f\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698488 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698449    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698510    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698608    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698664    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.698764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.698721    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.732668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.732634    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"]
Oct 30 20:14:54.826012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.825094    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.826291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.826266    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.826855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.826791    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.828589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.825967    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.828714 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.826695    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.828714 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.828449    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.828933 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.828902    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.829141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.829116    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpx5f\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.829269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.829205    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.829369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.829277    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.829427 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.829394    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.829483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.829457    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.830517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.829949    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.830517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.830040    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.836543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.834581    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.837971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.837944    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.916812 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.916714    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpx5f\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.930622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.930586    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token\") pod \"e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") " pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:54.938452 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:54.938420    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:55.418057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.418022    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"]
Oct 30 20:14:55.419365 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:14:55.419337    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93420d97_cb7f_4516_b7bd_75922905beb3.slice/crio-dda78a4071e72f6ce2cae9f4387d5395f179067ebec8e9455eb1386b2ecabe49 WatchSource:0}: Error finding container dda78a4071e72f6ce2cae9f4387d5395f179067ebec8e9455eb1386b2ecabe49: Status 404 returned error can't find the container with id dda78a4071e72f6ce2cae9f4387d5395f179067ebec8e9455eb1386b2ecabe49
Oct 30 20:14:55.422679 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.422637    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:14:55.422764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.422730    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:14:55.422831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.422813    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:14:55.569907 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.569868    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" event={"ID":"93420d97-cb7f-4516-b7bd-75922905beb3","Type":"ContainerStarted","Data":"9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"}
Oct 30 20:14:55.570075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.569933    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" event={"ID":"93420d97-cb7f-4516-b7bd-75922905beb3","Type":"ContainerStarted","Data":"dda78a4071e72f6ce2cae9f4387d5395f179067ebec8e9455eb1386b2ecabe49"}
Oct 30 20:14:55.700815 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.699813    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podStartSLOduration=1.6997951 podStartE2EDuration="1.6997951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:55.694956917 +0000 UTC m=+1991.859358560" watchObservedRunningTime="2026-10-30 20:14:55.6997951 +0000 UTC m=+1991.864196740"
Oct 30 20:14:55.941180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:55.941141    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:56.944679 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:56.944646    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:57.584246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:57.584215    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:57.585228 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:57.585201    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:14:59.738468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.738442    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"]
Oct 30 20:14:59.743114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.743090    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.797282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.797200    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:14:59.802645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.802619    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"]
Oct 30 20:14:59.807515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.807489    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.807658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.807554    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.807702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.807659    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqc9j\" (UniqueName: \"kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.847472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.847444    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:14:59.847992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.847958    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:59.847992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.847969    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:14:59.908817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.908758    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.908992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.908839    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.908992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.908864    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqc9j\" (UniqueName: \"kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.909521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.909495    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.912036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.912009    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:14:59.971374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:14:59.971341    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqc9j\" (UniqueName: \"kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j\") pod \"maas-api-revoke-keys-e2e-ait-577fb85a-55fxd\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:15:00.057703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.057623    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:15:00.185304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.185267    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"]
Oct 30 20:15:00.202728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.202698    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"]
Oct 30 20:15:00.206312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.202833    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.208491 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.208472    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"]
Oct 30 20:15:00.208605 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.208583    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.213179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.213152    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"]
Oct 30 20:15:00.213286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.213246    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.217524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.217505    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.232067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.232043    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"]
Oct 30 20:15:00.235979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.235961    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.272791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.268291    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"]
Oct 30 20:15:00.286092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.286044    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"]
Oct 30 20:15:00.286267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.286173    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.289860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.289831    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"]
Oct 30 20:15:00.306879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.306852    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"]
Oct 30 20:15:00.313109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.312831    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.314802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.314197    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"]
Oct 30 20:15:00.317884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.317861    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk9d\" (UniqueName: \"kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318053    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318133    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnj49\" (UniqueName: \"kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318179    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318204    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318223    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318266    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318289    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318308    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zgg\" (UniqueName: \"kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318349    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmbl\" (UniqueName: \"kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318376    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl9fb\" (UniqueName: \"kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.318573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.318448    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5v7n\" (UniqueName: \"kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.320093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.320072    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.320852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.320836    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"]
Oct 30 20:15:00.329152 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.327836    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"]
Oct 30 20:15:00.333711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.333693    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"]
Oct 30 20:15:00.340720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.340694    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"]
Oct 30 20:15:00.348411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.348388    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"]
Oct 30 20:15:00.348589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.348520    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.362141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.362066    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"]
Oct 30 20:15:00.372371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.372345    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"]
Oct 30 20:15:00.378099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.378073    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"]
Oct 30 20:15:00.420045 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420010    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnj49\" (UniqueName: \"kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.420205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420065    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.420205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420107    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.420205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420139    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.420411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420203    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.420411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420271    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.420411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420313    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.420411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420379    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.420411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420410    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zgg\" (UniqueName: \"kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.420676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420432    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85fv\" (UniqueName: \"kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.420676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420499    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmbl\" (UniqueName: \"kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.420676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420547    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9fb\" (UniqueName: \"kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.420676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420634    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.420941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420688    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5v7n\" (UniqueName: \"kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.420941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420750    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk9d\" (UniqueName: \"kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.420941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420830    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.420941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420859    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8655\" (UniqueName: \"kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655\") pod \"maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.421164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.420965    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr66c\" (UniqueName: \"kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.421164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.421113    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.421164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.421130    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.421164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.421135    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.421164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.421156    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.421381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.421307    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.421617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.421600    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.442476 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:00.442421    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf692ddd8_19ce_445a_b202_0a5825349763.slice/crio-416cc29fc40a5fb9fd92cb82f3002acd8f54a6077849b4ea3bad598aadbd9f1b WatchSource:0}: Error finding container 416cc29fc40a5fb9fd92cb82f3002acd8f54a6077849b4ea3bad598aadbd9f1b: Status 404 returned error can't find the container with id 416cc29fc40a5fb9fd92cb82f3002acd8f54a6077849b4ea3bad598aadbd9f1b
Oct 30 20:15:00.443700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.443671    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"]
Oct 30 20:15:00.482274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.482246    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmbl\" (UniqueName: \"kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.482448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.482417    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnj49\" (UniqueName: \"kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.483028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.483000    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9fb\" (UniqueName: \"kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.483105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.483053    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk9d\" (UniqueName: \"kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d\") pod \"maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.483640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.483621    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zgg\" (UniqueName: \"kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.484048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.484029    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5v7n\" (UniqueName: \"kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n\") pod \"maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.522393 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.522363    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.522550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.522472    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.522550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.522533    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85fv\" (UniqueName: \"kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.522701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.522676    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.522830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.522791    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8655\" (UniqueName: \"kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655\") pod \"maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.522958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.522873    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr66c\" (UniqueName: \"kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.523097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.523067    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.523300 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.523279    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.523381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.523371    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.523481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.523466    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:00.531983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.531962    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:00.540102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.540084    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:00.547356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.547334    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:00.559652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.559627    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:00.587999 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.587962    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85fv\" (UniqueName: \"kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:00.601227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.599871    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8655\" (UniqueName: \"kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655\") pod \"maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.622163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.621528    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:00.637499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.636929    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:00.646386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.640450    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr66c\" (UniqueName: \"kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.651539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.651410    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:00.664283 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.664232    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd" event={"ID":"f692ddd8-19ce-445a-b202-0a5825349763","Type":"ContainerStarted","Data":"416cc29fc40a5fb9fd92cb82f3002acd8f54a6077849b4ea3bad598aadbd9f1b"}
Oct 30 20:15:00.677153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:00.677123    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:01.128059 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:01.128012    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d915be_9593_42c8_9a13_4758f10a4e54.slice/crio-ab97159d6e12a147d6bf3e170b8e89c1261104de41f65fa96adebc2e97e38d3e WatchSource:0}: Error finding container ab97159d6e12a147d6bf3e170b8e89c1261104de41f65fa96adebc2e97e38d3e: Status 404 returned error can't find the container with id ab97159d6e12a147d6bf3e170b8e89c1261104de41f65fa96adebc2e97e38d3e
Oct 30 20:15:01.129398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.129366    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"]
Oct 30 20:15:01.177581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.177557    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"]
Oct 30 20:15:01.180856 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:01.179290    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd18c01_a9bb_4b82_a531_0734ae14e8ec.slice/crio-d45bca601515995c09b2679d456dfbcc33a95aeb56be1dc33c4b4b7b84d8e159 WatchSource:0}: Error finding container d45bca601515995c09b2679d456dfbcc33a95aeb56be1dc33c4b4b7b84d8e159: Status 404 returned error can't find the container with id d45bca601515995c09b2679d456dfbcc33a95aeb56be1dc33c4b4b7b84d8e159
Oct 30 20:15:01.193443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.192638    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"]
Oct 30 20:15:01.204213 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:01.204187    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f6b9e4_b141_4bd4_82fe_8630ee0cf295.slice/crio-06e1e4d2803beb5759d9fbaf50fbf6d5a5ac4be6512bac830e86067a1d34829a WatchSource:0}: Error finding container 06e1e4d2803beb5759d9fbaf50fbf6d5a5ac4be6512bac830e86067a1d34829a: Status 404 returned error can't find the container with id 06e1e4d2803beb5759d9fbaf50fbf6d5a5ac4be6512bac830e86067a1d34829a
Oct 30 20:15:01.228753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.228723    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"]
Oct 30 20:15:01.229991 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:01.229964    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c35eac_3545_4cd4_a171_8a75585aeaa4.slice/crio-eb5a7be7fdfc98775f195a42726dbfe6cb14c3a782ad948f1f3156e227160b8c WatchSource:0}: Error finding container eb5a7be7fdfc98775f195a42726dbfe6cb14c3a782ad948f1f3156e227160b8c: Status 404 returned error can't find the container with id eb5a7be7fdfc98775f195a42726dbfe6cb14c3a782ad948f1f3156e227160b8c
Oct 30 20:15:01.258972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.258945    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"]
Oct 30 20:15:01.305657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.305627    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"]
Oct 30 20:15:01.337823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.337791    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"]
Oct 30 20:15:01.345322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.345295    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"]
Oct 30 20:15:01.351453 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:01.351428    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53236041_b9c6_4c73_a1b5_165288c1923b.slice/crio-14b7054d245395fd3afd9161eac311fc717b6db2ef7064bb86d54fc1f6b2f462 WatchSource:0}: Error finding container 14b7054d245395fd3afd9161eac311fc717b6db2ef7064bb86d54fc1f6b2f462: Status 404 returned error can't find the container with id 14b7054d245395fd3afd9161eac311fc717b6db2ef7064bb86d54fc1f6b2f462
Oct 30 20:15:01.475802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.475757    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"]
Oct 30 20:15:01.549397 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:01.549367    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701b2111_b1dd_48d1_bd36_dc379850598c.slice/crio-0ed5f974e22bc2e3a0a35efa197bb99fbaa012d951f5735a9a7bf85731e4bbd2 WatchSource:0}: Error finding container 0ed5f974e22bc2e3a0a35efa197bb99fbaa012d951f5735a9a7bf85731e4bbd2: Status 404 returned error can't find the container with id 0ed5f974e22bc2e3a0a35efa197bb99fbaa012d951f5735a9a7bf85731e4bbd2
Oct 30 20:15:01.673880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.673850    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerStarted","Data":"eb5a7be7fdfc98775f195a42726dbfe6cb14c3a782ad948f1f3156e227160b8c"}
Oct 30 20:15:01.675445 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.675420    4917 generic.go:356] "Generic (PLEG): container finished" podID="f692ddd8-19ce-445a-b202-0a5825349763" containerID="966103cf70948c5d244d05dafd191e72b34abdc6e5cf80caa4d31590d2f283e0" exitCode=0
Oct 30 20:15:01.675535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.675499    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd" event={"ID":"f692ddd8-19ce-445a-b202-0a5825349763","Type":"ContainerDied","Data":"966103cf70948c5d244d05dafd191e72b34abdc6e5cf80caa4d31590d2f283e0"}
Oct 30 20:15:01.676926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.676906    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" event={"ID":"d8dbd364-a935-4fac-8d75-71a6e93e7f58","Type":"ContainerStarted","Data":"0b22648953181dc7eb834e7b990b91e491fda877b1bd8e660112f9fee4fea1cb"}
Oct 30 20:15:01.678156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.678138    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" event={"ID":"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295","Type":"ContainerStarted","Data":"06e1e4d2803beb5759d9fbaf50fbf6d5a5ac4be6512bac830e86067a1d34829a"}
Oct 30 20:15:01.679333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.679311    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerStarted","Data":"14b7054d245395fd3afd9161eac311fc717b6db2ef7064bb86d54fc1f6b2f462"}
Oct 30 20:15:01.680552 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.680530    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" event={"ID":"5e3f9666-894e-4fdc-b378-b4b33ed8d562","Type":"ContainerStarted","Data":"8def2c13af78090069b0b6bea8cd5d1c4c1c31ea2cd5a4bfad34aeb4cb205d4a"}
Oct 30 20:15:01.681662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.681635    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerStarted","Data":"0ed5f974e22bc2e3a0a35efa197bb99fbaa012d951f5735a9a7bf85731e4bbd2"}
Oct 30 20:15:01.682936 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.682917    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" event={"ID":"1ee786e2-f600-4d0a-8853-15f61a5ac87b","Type":"ContainerStarted","Data":"ba1745de2ee6020d44e67c4157b2fa7e08fa0e1251abd7d9c7213eb4882c78a8"}
Oct 30 20:15:01.684604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.684584    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerStarted","Data":"a227d6934b7b3608f1249de60191d8f7dbfaacbbb3c73213c92649e8ae770558"}
Oct 30 20:15:01.684703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.684633    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerStarted","Data":"ab97159d6e12a147d6bf3e170b8e89c1261104de41f65fa96adebc2e97e38d3e"}
Oct 30 20:15:01.686214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.686178    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerStarted","Data":"db71610e5873c66de39b9f4d55bc9be2a38fac8899d98ae00127a8ae25d38f0c"}
Oct 30 20:15:01.686301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.686220    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerStarted","Data":"d45bca601515995c09b2679d456dfbcc33a95aeb56be1dc33c4b4b7b84d8e159"}
Oct 30 20:15:01.932523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:01.932467    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" podStartSLOduration=1.932450623 podStartE2EDuration="1.932450623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.931538658 +0000 UTC m=+1998.095940302" watchObservedRunningTime="2026-10-30 20:15:01.932450623 +0000 UTC m=+1998.096852264"
Oct 30 20:15:02.056559 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.056501    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" podStartSLOduration=2.056485654 podStartE2EDuration="2.056485654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.055136096 +0000 UTC m=+1998.219537750" watchObservedRunningTime="2026-10-30 20:15:02.056485654 +0000 UTC m=+1998.220887295"
Oct 30 20:15:02.556674 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:02.556588    4917 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f6b9e4_b141_4bd4_82fe_8630ee0cf295.slice/crio-conmon-cb5fe6d9884c38c94a4d8743135647d6b4d1fdc589144a3bded2fe5c53298593.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dbd364_a935_4fac_8d75_71a6e93e7f58.slice/crio-conmon-655f66baa75ca39e9d5f8fe4c62d842f4844184a0e469ebfaff4dbcb85dcc7db.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:15:02.701965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.701914    4917 generic.go:356] "Generic (PLEG): container finished" podID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerID="75a881ccf608e833bc2f8b99331a8705e4b0012ca3deae6497bff559f087293f" exitCode=60
Oct 30 20:15:02.702543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.702277    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" event={"ID":"5e3f9666-894e-4fdc-b378-b4b33ed8d562","Type":"ContainerDied","Data":"75a881ccf608e833bc2f8b99331a8705e4b0012ca3deae6497bff559f087293f"}
Oct 30 20:15:02.703027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.702999    4917 scope.go:122] "RemoveContainer" containerID="75a881ccf608e833bc2f8b99331a8705e4b0012ca3deae6497bff559f087293f"
Oct 30 20:15:02.705302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.705274    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerStarted","Data":"045c618c5bd7cbf1992d666e59fb19d58c1ae4595a4938c128ff216302f742d7"}
Oct 30 20:15:02.731951 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.731763    4917 generic.go:356] "Generic (PLEG): container finished" podID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerID="c690089a8fd70e2718e60077201974b091313e74a7179f094ba3fc2ca9566eb4" exitCode=60
Oct 30 20:15:02.732144 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.732101    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" event={"ID":"1ee786e2-f600-4d0a-8853-15f61a5ac87b","Type":"ContainerDied","Data":"c690089a8fd70e2718e60077201974b091313e74a7179f094ba3fc2ca9566eb4"}
Oct 30 20:15:02.732528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.732503    4917 scope.go:122] "RemoveContainer" containerID="c690089a8fd70e2718e60077201974b091313e74a7179f094ba3fc2ca9566eb4"
Oct 30 20:15:02.739301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.737906    4917 generic.go:356] "Generic (PLEG): container finished" podID="67d915be-9593-42c8-9a13-4758f10a4e54" containerID="a227d6934b7b3608f1249de60191d8f7dbfaacbbb3c73213c92649e8ae770558" exitCode=60
Oct 30 20:15:02.739301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.738016    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerDied","Data":"a227d6934b7b3608f1249de60191d8f7dbfaacbbb3c73213c92649e8ae770558"}
Oct 30 20:15:02.739301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.738307    4917 scope.go:122] "RemoveContainer" containerID="a227d6934b7b3608f1249de60191d8f7dbfaacbbb3c73213c92649e8ae770558"
Oct 30 20:15:02.757878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.757848    4917 generic.go:356] "Generic (PLEG): container finished" podID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerID="db71610e5873c66de39b9f4d55bc9be2a38fac8899d98ae00127a8ae25d38f0c" exitCode=60
Oct 30 20:15:02.758189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.758165    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerDied","Data":"db71610e5873c66de39b9f4d55bc9be2a38fac8899d98ae00127a8ae25d38f0c"}
Oct 30 20:15:02.758817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.758796    4917 scope.go:122] "RemoveContainer" containerID="db71610e5873c66de39b9f4d55bc9be2a38fac8899d98ae00127a8ae25d38f0c"
Oct 30 20:15:02.765051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.764984    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerStarted","Data":"d5e91e3c8dbf832e91e9adf4ac53799bdb7c360b5293e5f372ccaefd47f20baf"}
Oct 30 20:15:02.768426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.768200    4917 generic.go:356] "Generic (PLEG): container finished" podID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerID="655f66baa75ca39e9d5f8fe4c62d842f4844184a0e469ebfaff4dbcb85dcc7db" exitCode=60
Oct 30 20:15:02.768426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.768321    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" event={"ID":"d8dbd364-a935-4fac-8d75-71a6e93e7f58","Type":"ContainerDied","Data":"655f66baa75ca39e9d5f8fe4c62d842f4844184a0e469ebfaff4dbcb85dcc7db"}
Oct 30 20:15:02.769136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.768789    4917 scope.go:122] "RemoveContainer" containerID="655f66baa75ca39e9d5f8fe4c62d842f4844184a0e469ebfaff4dbcb85dcc7db"
Oct 30 20:15:02.815274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.814348    4917 generic.go:356] "Generic (PLEG): container finished" podID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerID="cb5fe6d9884c38c94a4d8743135647d6b4d1fdc589144a3bded2fe5c53298593" exitCode=60
Oct 30 20:15:02.815274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.814691    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" event={"ID":"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295","Type":"ContainerDied","Data":"cb5fe6d9884c38c94a4d8743135647d6b4d1fdc589144a3bded2fe5c53298593"}
Oct 30 20:15:02.815274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.814890    4917 scope.go:122] "RemoveContainer" containerID="cb5fe6d9884c38c94a4d8743135647d6b4d1fdc589144a3bded2fe5c53298593"
Oct 30 20:15:02.818269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.817513    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerStarted","Data":"e3483c59d04a1093afae3aa8a0783efa9737f09205767ce87ec07fb46a30d31d"}
Oct 30 20:15:02.818269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:02.817852    4917 scope.go:122] "RemoveContainer" containerID="e3483c59d04a1093afae3aa8a0783efa9737f09205767ce87ec07fb46a30d31d"
Oct 30 20:15:03.172791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.169488    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" podStartSLOduration=3.169463905 podStartE2EDuration="3.169463905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.090854838 +0000 UTC m=+1999.255256480" watchObservedRunningTime="2026-10-30 20:15:03.169463905 +0000 UTC m=+1999.333865550"
Oct 30 20:15:03.262884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.262758    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" podStartSLOduration=3.2625970779999998 podStartE2EDuration="3.262597078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.220298257 +0000 UTC m=+1999.384699900" watchObservedRunningTime="2026-10-30 20:15:03.262597078 +0000 UTC m=+1999.426998721"
Oct 30 20:15:03.290076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.290050    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:15:03.433793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.433748    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j\" (UniqueName: \"kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j\") pod \"f692ddd8-19ce-445a-b202-0a5825349763\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") "
Oct 30 20:15:03.434021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.434001    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca\") pod \"f692ddd8-19ce-445a-b202-0a5825349763\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") "
Oct 30 20:15:03.434104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.434056    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca\") pod \"f692ddd8-19ce-445a-b202-0a5825349763\" (UID: \"f692ddd8-19ce-445a-b202-0a5825349763\") "
Oct 30 20:15:03.434427 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.434395    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca" pod "f692ddd8-19ce-445a-b202-0a5825349763" (UID: "f692ddd8-19ce-445a-b202-0a5825349763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:03.434800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.434749    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:03.436201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.436177    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j" pod "f692ddd8-19ce-445a-b202-0a5825349763" (UID: "f692ddd8-19ce-445a-b202-0a5825349763"). InnerVolumeSpecName "kube-api-access-gqc9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:03.436327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.436308    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca" pod "f692ddd8-19ce-445a-b202-0a5825349763" (UID: "f692ddd8-19ce-445a-b202-0a5825349763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:03.536486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.536449    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f692ddd8-19ce-445a-b202-0a5825349763-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:03.536486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.536488    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqc9j\" (UniqueName: \"kubernetes.io/projected/f692ddd8-19ce-445a-b202-0a5825349763-kube-api-access-gqc9j\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:03.849255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.849166    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"
Oct 30 20:15:03.849255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.849174    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd" event={"ID":"f692ddd8-19ce-445a-b202-0a5825349763","Type":"ContainerDied","Data":"416cc29fc40a5fb9fd92cb82f3002acd8f54a6077849b4ea3bad598aadbd9f1b"}
Oct 30 20:15:03.849814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.849303    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416cc29fc40a5fb9fd92cb82f3002acd8f54a6077849b4ea3bad598aadbd9f1b"
Oct 30 20:15:03.852835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.852795    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" event={"ID":"d8dbd364-a935-4fac-8d75-71a6e93e7f58","Type":"ContainerStarted","Data":"fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc"}
Oct 30 20:15:03.855428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.855399    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" event={"ID":"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295","Type":"ContainerStarted","Data":"8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028"}
Oct 30 20:15:03.857580 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.857555    4917 generic.go:356] "Generic (PLEG): container finished" podID="53236041-b9c6-4c73-a1b5-165288c1923b" containerID="e3483c59d04a1093afae3aa8a0783efa9737f09205767ce87ec07fb46a30d31d" exitCode=60
Oct 30 20:15:03.857690 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.857636    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerDied","Data":"e3483c59d04a1093afae3aa8a0783efa9737f09205767ce87ec07fb46a30d31d"}
Oct 30 20:15:03.857754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.857689    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerStarted","Data":"b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103"}
Oct 30 20:15:03.860112 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.860089    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" event={"ID":"5e3f9666-894e-4fdc-b378-b4b33ed8d562","Type":"ContainerStarted","Data":"abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750"}
Oct 30 20:15:03.861934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.861906    4917 generic.go:356] "Generic (PLEG): container finished" podID="701b2111-b1dd-48d1-bd36-dc379850598c" containerID="045c618c5bd7cbf1992d666e59fb19d58c1ae4595a4938c128ff216302f742d7" exitCode=60
Oct 30 20:15:03.862393 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.862002    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerDied","Data":"045c618c5bd7cbf1992d666e59fb19d58c1ae4595a4938c128ff216302f742d7"}
Oct 30 20:15:03.862393 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.862276    4917 scope.go:122] "RemoveContainer" containerID="045c618c5bd7cbf1992d666e59fb19d58c1ae4595a4938c128ff216302f742d7"
Oct 30 20:15:03.864976 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.864950    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" event={"ID":"1ee786e2-f600-4d0a-8853-15f61a5ac87b","Type":"ContainerStarted","Data":"b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442"}
Oct 30 20:15:03.867485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.867461    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerStarted","Data":"e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b"}
Oct 30 20:15:03.869892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.869870    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerStarted","Data":"39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755"}
Oct 30 20:15:03.872541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.871763    4917 generic.go:356] "Generic (PLEG): container finished" podID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerID="d5e91e3c8dbf832e91e9adf4ac53799bdb7c360b5293e5f372ccaefd47f20baf" exitCode=60
Oct 30 20:15:03.872541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.871804    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerDied","Data":"d5e91e3c8dbf832e91e9adf4ac53799bdb7c360b5293e5f372ccaefd47f20baf"}
Oct 30 20:15:03.872541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:03.872123    4917 scope.go:122] "RemoveContainer" containerID="d5e91e3c8dbf832e91e9adf4ac53799bdb7c360b5293e5f372ccaefd47f20baf"
Oct 30 20:15:04.195835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.195746    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" podStartSLOduration=4.195731538 podStartE2EDuration="4.195731538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.194255728 +0000 UTC m=+2000.358657370" watchObservedRunningTime="2026-10-30 20:15:04.195731538 +0000 UTC m=+2000.360133179"
Oct 30 20:15:04.431268 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.431217    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" podStartSLOduration=4.431203354 podStartE2EDuration="4.431203354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.428733619 +0000 UTC m=+2000.593135263" watchObservedRunningTime="2026-10-30 20:15:04.431203354 +0000 UTC m=+2000.595605035"
Oct 30 20:15:04.724782 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.724706    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" podStartSLOduration=4.724691547 podStartE2EDuration="4.724691547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.548828281 +0000 UTC m=+2000.713229925" watchObservedRunningTime="2026-10-30 20:15:04.724691547 +0000 UTC m=+2000.889093189"
Oct 30 20:15:04.886872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.886829    4917 generic.go:356] "Generic (PLEG): container finished" podID="53236041-b9c6-4c73-a1b5-165288c1923b" containerID="b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103" exitCode=60
Oct 30 20:15:04.887374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.886919    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerDied","Data":"b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103"}
Oct 30 20:15:04.887374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.886991    4917 scope.go:122] "RemoveContainer" containerID="e3483c59d04a1093afae3aa8a0783efa9737f09205767ce87ec07fb46a30d31d"
Oct 30 20:15:04.887374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.887354    4917 scope.go:122] "RemoveContainer" containerID="b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103"
Oct 30 20:15:04.887878 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.887851    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r_odh-ai-gateway-infra(53236041-b9c6-4c73-a1b5-165288c1923b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" podUID="53236041-b9c6-4c73-a1b5-165288c1923b"
Oct 30 20:15:04.890365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.890328    4917 generic.go:356] "Generic (PLEG): container finished" podID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerID="abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750" exitCode=60
Oct 30 20:15:04.890553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.890378    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" event={"ID":"5e3f9666-894e-4fdc-b378-b4b33ed8d562","Type":"ContainerDied","Data":"abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750"}
Oct 30 20:15:04.890817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.890783    4917 scope.go:122] "RemoveContainer" containerID="abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750"
Oct 30 20:15:04.891276 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.891252    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck_odh-ai-gateway-infra(5e3f9666-894e-4fdc-b378-b4b33ed8d562)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562"
Oct 30 20:15:04.893113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.892717    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerStarted","Data":"a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9"}
Oct 30 20:15:04.895220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.895201    4917 generic.go:356] "Generic (PLEG): container finished" podID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerID="b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442" exitCode=60
Oct 30 20:15:04.895320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.895274    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" event={"ID":"1ee786e2-f600-4d0a-8853-15f61a5ac87b","Type":"ContainerDied","Data":"b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442"}
Oct 30 20:15:04.895506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.895491    4917 scope.go:122] "RemoveContainer" containerID="b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442"
Oct 30 20:15:04.895847 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.895823    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r_odh-ai-gateway-infra(1ee786e2-f600-4d0a-8853-15f61a5ac87b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b"
Oct 30 20:15:04.897525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.897501    4917 generic.go:356] "Generic (PLEG): container finished" podID="67d915be-9593-42c8-9a13-4758f10a4e54" containerID="e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b" exitCode=60
Oct 30 20:15:04.897615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.897575    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerDied","Data":"e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b"}
Oct 30 20:15:04.897885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.897867    4917 scope.go:122] "RemoveContainer" containerID="e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b"
Oct 30 20:15:04.898205 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.898183    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv_odh-ai-gateway-infra(67d915be-9593-42c8-9a13-4758f10a4e54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" podUID="67d915be-9593-42c8-9a13-4758f10a4e54"
Oct 30 20:15:04.899460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.899439    4917 generic.go:356] "Generic (PLEG): container finished" podID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerID="39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755" exitCode=60
Oct 30 20:15:04.899561 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.899521    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerDied","Data":"39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755"}
Oct 30 20:15:04.899757 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.899741    4917 scope.go:122] "RemoveContainer" containerID="39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755"
Oct 30 20:15:04.900062 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.900039    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6_odh-ai-gateway-infra(7cd18c01-a9bb-4b82-a531-0734ae14e8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec"
Oct 30 20:15:04.901807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.901731    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerStarted","Data":"d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c"}
Oct 30 20:15:04.903694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.903676    4917 generic.go:356] "Generic (PLEG): container finished" podID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerID="fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc" exitCode=60
Oct 30 20:15:04.903846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.903756    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" event={"ID":"d8dbd364-a935-4fac-8d75-71a6e93e7f58","Type":"ContainerDied","Data":"fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc"}
Oct 30 20:15:04.903959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.903944    4917 scope.go:122] "RemoveContainer" containerID="fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc"
Oct 30 20:15:04.904115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.904097    4917 scope.go:122] "RemoveContainer" containerID="75a881ccf608e833bc2f8b99331a8705e4b0012ca3deae6497bff559f087293f"
Oct 30 20:15:04.904189 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.904172    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb_odh-ai-gateway-infra(d8dbd364-a935-4fac-8d75-71a6e93e7f58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58"
Oct 30 20:15:04.905833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.905812    4917 generic.go:356] "Generic (PLEG): container finished" podID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerID="8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028" exitCode=60
Oct 30 20:15:04.905920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.905844    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" event={"ID":"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295","Type":"ContainerDied","Data":"8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028"}
Oct 30 20:15:04.906111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.906099    4917 scope.go:122] "RemoveContainer" containerID="8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028"
Oct 30 20:15:04.906319 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:04.906303    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8_odh-ai-gateway-infra(e9f6b9e4-b141-4bd4-82fe-8630ee0cf295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295"
Oct 30 20:15:04.918322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.918299    4917 scope.go:122] "RemoveContainer" containerID="c690089a8fd70e2718e60077201974b091313e74a7179f094ba3fc2ca9566eb4"
Oct 30 20:15:04.937188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.937162    4917 scope.go:122] "RemoveContainer" containerID="a227d6934b7b3608f1249de60191d8f7dbfaacbbb3c73213c92649e8ae770558"
Oct 30 20:15:04.956322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.956272    4917 scope.go:122] "RemoveContainer" containerID="db71610e5873c66de39b9f4d55bc9be2a38fac8899d98ae00127a8ae25d38f0c"
Oct 30 20:15:04.972244 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.971847    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" podStartSLOduration=4.971827081 podStartE2EDuration="4.971827081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.964224178 +0000 UTC m=+2001.128625818" watchObservedRunningTime="2026-10-30 20:15:04.971827081 +0000 UTC m=+2001.136228723"
Oct 30 20:15:04.985109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:04.985081    4917 scope.go:122] "RemoveContainer" containerID="655f66baa75ca39e9d5f8fe4c62d842f4844184a0e469ebfaff4dbcb85dcc7db"
Oct 30 20:15:05.018970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.018940    4917 scope.go:122] "RemoveContainer" containerID="cb5fe6d9884c38c94a4d8743135647d6b4d1fdc589144a3bded2fe5c53298593"
Oct 30 20:15:05.085991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.085930    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" podStartSLOduration=5.085912644 podStartE2EDuration="5.085912644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:05.085177845 +0000 UTC m=+2001.249579487" watchObservedRunningTime="2026-10-30 20:15:05.085912644 +0000 UTC m=+2001.250314287"
Oct 30 20:15:05.916747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.916712    4917 scope.go:122] "RemoveContainer" containerID="fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc"
Oct 30 20:15:05.917235 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.917100    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb_odh-ai-gateway-infra(d8dbd364-a935-4fac-8d75-71a6e93e7f58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58"
Oct 30 20:15:05.918497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.918474    4917 scope.go:122] "RemoveContainer" containerID="8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028"
Oct 30 20:15:05.918870 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.918845    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8_odh-ai-gateway-infra(e9f6b9e4-b141-4bd4-82fe-8630ee0cf295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295"
Oct 30 20:15:05.920038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.920020    4917 scope.go:122] "RemoveContainer" containerID="b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103"
Oct 30 20:15:05.920345 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.920324    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r_odh-ai-gateway-infra(53236041-b9c6-4c73-a1b5-165288c1923b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" podUID="53236041-b9c6-4c73-a1b5-165288c1923b"
Oct 30 20:15:05.921534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.921508    4917 scope.go:122] "RemoveContainer" containerID="abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750"
Oct 30 20:15:05.921813 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.921793    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck_odh-ai-gateway-infra(5e3f9666-894e-4fdc-b378-b4b33ed8d562)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562"
Oct 30 20:15:05.923106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.923084    4917 generic.go:356] "Generic (PLEG): container finished" podID="701b2111-b1dd-48d1-bd36-dc379850598c" containerID="a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9" exitCode=60
Oct 30 20:15:05.923202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.923153    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerDied","Data":"a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9"}
Oct 30 20:15:05.923265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.923239    4917 scope.go:122] "RemoveContainer" containerID="045c618c5bd7cbf1992d666e59fb19d58c1ae4595a4938c128ff216302f742d7"
Oct 30 20:15:05.923400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.923384    4917 scope.go:122] "RemoveContainer" containerID="a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9"
Oct 30 20:15:05.923644 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.923625    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756_odh-ai-gateway-infra(701b2111-b1dd-48d1-bd36-dc379850598c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" podUID="701b2111-b1dd-48d1-bd36-dc379850598c"
Oct 30 20:15:05.924955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.924939    4917 scope.go:122] "RemoveContainer" containerID="b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442"
Oct 30 20:15:05.925197 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.925173    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r_odh-ai-gateway-infra(1ee786e2-f600-4d0a-8853-15f61a5ac87b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b"
Oct 30 20:15:05.926442 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.926426    4917 scope.go:122] "RemoveContainer" containerID="e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b"
Oct 30 20:15:05.926662 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.926642    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv_odh-ai-gateway-infra(67d915be-9593-42c8-9a13-4758f10a4e54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" podUID="67d915be-9593-42c8-9a13-4758f10a4e54"
Oct 30 20:15:05.927996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.927978    4917 scope.go:122] "RemoveContainer" containerID="39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755"
Oct 30 20:15:05.928192 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.928172    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6_odh-ai-gateway-infra(7cd18c01-a9bb-4b82-a531-0734ae14e8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec"
Oct 30 20:15:05.930197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.930174    4917 generic.go:356] "Generic (PLEG): container finished" podID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerID="d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c" exitCode=60
Oct 30 20:15:05.930342 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.930256    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerDied","Data":"d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c"}
Oct 30 20:15:05.930492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.930474    4917 scope.go:122] "RemoveContainer" containerID="d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c"
Oct 30 20:15:05.930780 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:05.930744    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl_odh-ai-gateway-infra(21c35eac-3545-4cd4-a171-8a75585aeaa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4"
Oct 30 20:15:05.938640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:05.938621    4917 scope.go:122] "RemoveContainer" containerID="d5e91e3c8dbf832e91e9adf4ac53799bdb7c360b5293e5f372ccaefd47f20baf"
Oct 30 20:15:06.943444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:06.943407    4917 scope.go:122] "RemoveContainer" containerID="d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c"
Oct 30 20:15:06.943961 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:06.943857    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl_odh-ai-gateway-infra(21c35eac-3545-4cd4-a171-8a75585aeaa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4"
Oct 30 20:15:06.946476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:06.946445    4917 scope.go:122] "RemoveContainer" containerID="a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9"
Oct 30 20:15:06.946873 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:06.946839    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756_odh-ai-gateway-infra(701b2111-b1dd-48d1-bd36-dc379850598c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" podUID="701b2111-b1dd-48d1-bd36-dc379850598c"
Oct 30 20:15:09.680205 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:09.680167    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:15:12.959889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:12.959849    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"]
Oct 30 20:15:12.960969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:12.960943    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f692ddd8-19ce-445a-b202-0a5825349763" containerName="revoke-keys"
Oct 30 20:15:12.960969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:12.960966    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f692ddd8-19ce-445a-b202-0a5825349763" containerName="revoke-keys"
Oct 30 20:15:12.961156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:12.961120    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f692ddd8-19ce-445a-b202-0a5825349763" containerName="revoke-keys"
Oct 30 20:15:12.964815 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:12.964783    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.026010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.025972    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"]
Oct 30 20:15:13.066560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.066531    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-37732de7\"/\"openshift-service-ca.crt\""
Oct 30 20:15:13.066752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.066721    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-37732de7\"/\"e2e-lifecycle-model-37732de7-kserve-self-signed-certs\""
Oct 30 20:15:13.066875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.066831    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-37732de7\"/\"default-dockercfg-pbjfh\""
Oct 30 20:15:13.069631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.069610    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-37732de7\"/\"kube-root-ca.crt\""
Oct 30 20:15:13.103620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.103590    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.103748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.103666    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.103923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.103902    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.103994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.103935    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.103994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.103970    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.104070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.104047    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znllk\" (UniqueName: \"kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.211732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.211627    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.211732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.211692    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.212006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.211755    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.212006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.211835    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znllk\" (UniqueName: \"kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.212006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.211968    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.212164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.212084    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.217462 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.217427    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.217639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.217509    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.218018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.217990    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.219857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.219831    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.220605 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.220581    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.297339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.297304    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znllk\" (UniqueName: \"kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk\") pod \"e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") " pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:13.598856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:13.598444    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:14.064989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:14.064955    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:15:14.177556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:14.177526    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"]
Oct 30 20:15:14.177944 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:15:14.177921    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe8fcdc8_23a3_4899_8351_56de8211847f.slice/crio-6064b7861e71c629574f8f512955f4f72bcf090535bbd11236a240958a35a9cc WatchSource:0}: Error finding container 6064b7861e71c629574f8f512955f4f72bcf090535bbd11236a240958a35a9cc: Status 404 returned error can't find the container with id 6064b7861e71c629574f8f512955f4f72bcf090535bbd11236a240958a35a9cc
Oct 30 20:15:15.273934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.271249    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" event={"ID":"be8fcdc8-23a3-4899-8351-56de8211847f","Type":"ContainerStarted","Data":"0af1d2e5ff639399b77154666771b182f0b877c2ec0fb4bed93fe8f41be3a016"}
Oct 30 20:15:15.273934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.271299    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" event={"ID":"be8fcdc8-23a3-4899-8351-56de8211847f","Type":"ContainerStarted","Data":"6064b7861e71c629574f8f512955f4f72bcf090535bbd11236a240958a35a9cc"}
Oct 30 20:15:15.392096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.392028    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" podStartSLOduration=3.392014687 podStartE2EDuration="3.392014687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:15.391105096 +0000 UTC m=+2011.555506738" watchObservedRunningTime="2026-10-30 20:15:15.392014687 +0000 UTC m=+2011.556416328"
Oct 30 20:15:15.601861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.601764    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:15.995965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.995932    4917 scope.go:122] "RemoveContainer" containerID="abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750"
Oct 30 20:15:15.996575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.996548    4917 scope.go:122] "RemoveContainer" containerID="b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442"
Oct 30 20:15:15.996995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.996975    4917 scope.go:122] "RemoveContainer" containerID="39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755"
Oct 30 20:15:15.997504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.997484    4917 scope.go:122] "RemoveContainer" containerID="b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103"
Oct 30 20:15:15.997682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:15.997659    4917 scope.go:122] "RemoveContainer" containerID="e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b"
Oct 30 20:15:16.996040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:16.996004    4917 scope.go:122] "RemoveContainer" containerID="8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028"
Oct 30 20:15:16.996449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:16.996073    4917 scope.go:122] "RemoveContainer" containerID="a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9"
Oct 30 20:15:17.291756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.291730    4917 generic.go:356] "Generic (PLEG): container finished" podID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerID="6baf908c30aa0f1b0d40cc95e02dad2f8ccf6b231ebf5eecba718909fe394a44" exitCode=60
Oct 30 20:15:17.291915 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.291817    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" event={"ID":"1ee786e2-f600-4d0a-8853-15f61a5ac87b","Type":"ContainerDied","Data":"6baf908c30aa0f1b0d40cc95e02dad2f8ccf6b231ebf5eecba718909fe394a44"}
Oct 30 20:15:17.291915 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.291888    4917 scope.go:122] "RemoveContainer" containerID="b54ba99c83732ac0e615802f2d69e1d6b92c1a2e94c827ac02e88a6697d26442"
Oct 30 20:15:17.292376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.292348    4917 scope.go:122] "RemoveContainer" containerID="6baf908c30aa0f1b0d40cc95e02dad2f8ccf6b231ebf5eecba718909fe394a44"
Oct 30 20:15:17.292882 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:17.292814    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r_odh-ai-gateway-infra(1ee786e2-f600-4d0a-8853-15f61a5ac87b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b"
Oct 30 20:15:17.293914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.293893    4917 generic.go:356] "Generic (PLEG): container finished" podID="67d915be-9593-42c8-9a13-4758f10a4e54" containerID="f734ca68c3842722a6df073e84a184c60360a8bff4f4245f9cf6d00923a60aa1" exitCode=60
Oct 30 20:15:17.294006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.293969    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerDied","Data":"f734ca68c3842722a6df073e84a184c60360a8bff4f4245f9cf6d00923a60aa1"}
Oct 30 20:15:17.294426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.294406    4917 scope.go:122] "RemoveContainer" containerID="f734ca68c3842722a6df073e84a184c60360a8bff4f4245f9cf6d00923a60aa1"
Oct 30 20:15:17.294823 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:17.294762    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv_odh-ai-gateway-infra(67d915be-9593-42c8-9a13-4758f10a4e54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" podUID="67d915be-9593-42c8-9a13-4758f10a4e54"
Oct 30 20:15:17.295980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.295961    4917 generic.go:356] "Generic (PLEG): container finished" podID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerID="9ccd59ece7f79ff98bf3af3a78db0498c48f219f95ea3d2c67409608a5780d4d" exitCode=60
Oct 30 20:15:17.296084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.296035    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerDied","Data":"9ccd59ece7f79ff98bf3af3a78db0498c48f219f95ea3d2c67409608a5780d4d"}
Oct 30 20:15:17.296444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.296428    4917 scope.go:122] "RemoveContainer" containerID="9ccd59ece7f79ff98bf3af3a78db0498c48f219f95ea3d2c67409608a5780d4d"
Oct 30 20:15:17.296764 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:17.296744    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6_odh-ai-gateway-infra(7cd18c01-a9bb-4b82-a531-0734ae14e8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec"
Oct 30 20:15:17.298202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.298185    4917 generic.go:356] "Generic (PLEG): container finished" podID="53236041-b9c6-4c73-a1b5-165288c1923b" containerID="1b3e9024e32c75e5e92fdfb7ec40218a110835376bc293e4c3470f97ecfeb9d1" exitCode=60
Oct 30 20:15:17.298300 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.298241    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerDied","Data":"1b3e9024e32c75e5e92fdfb7ec40218a110835376bc293e4c3470f97ecfeb9d1"}
Oct 30 20:15:17.298607 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.298589    4917 scope.go:122] "RemoveContainer" containerID="1b3e9024e32c75e5e92fdfb7ec40218a110835376bc293e4c3470f97ecfeb9d1"
Oct 30 20:15:17.299068 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:17.299044    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r_odh-ai-gateway-infra(53236041-b9c6-4c73-a1b5-165288c1923b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" podUID="53236041-b9c6-4c73-a1b5-165288c1923b"
Oct 30 20:15:17.300316 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.300298    4917 generic.go:356] "Generic (PLEG): container finished" podID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerID="8a18e82a93c02123c029b8f63caa6281979492e0ddc3db04d7c33c53d2f24377" exitCode=60
Oct 30 20:15:17.300414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.300366    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" event={"ID":"5e3f9666-894e-4fdc-b378-b4b33ed8d562","Type":"ContainerDied","Data":"8a18e82a93c02123c029b8f63caa6281979492e0ddc3db04d7c33c53d2f24377"}
Oct 30 20:15:17.300693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.300680    4917 scope.go:122] "RemoveContainer" containerID="8a18e82a93c02123c029b8f63caa6281979492e0ddc3db04d7c33c53d2f24377"
Oct 30 20:15:17.300955 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:17.300934    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck_odh-ai-gateway-infra(5e3f9666-894e-4fdc-b378-b4b33ed8d562)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562"
Oct 30 20:15:17.519281 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.519257    4917 scope.go:122] "RemoveContainer" containerID="e99102476e933dadcc281c09104f0267b46cf8575a48dedfa0c684a6a300d47b"
Oct 30 20:15:17.537576 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.537554    4917 scope.go:122] "RemoveContainer" containerID="39ca4f3cad6107b166e85b0923d88d1f90c3fb7b470b59794be4967284376755"
Oct 30 20:15:17.552533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.552493    4917 scope.go:122] "RemoveContainer" containerID="b46ab2c3460934e34b69c20956273e647ca55f930b13f5a555df51e8d28d7103"
Oct 30 20:15:17.565889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:17.565859    4917 scope.go:122] "RemoveContainer" containerID="abcedb70fd1023b45296c4c7641a5f9a0b85e4343304b67c345261ce64dea750"
Oct 30 20:15:18.312574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.312496    4917 generic.go:356] "Generic (PLEG): container finished" podID="701b2111-b1dd-48d1-bd36-dc379850598c" containerID="1e1740849c8417d9da75d4fb00d1962e2cc4ecdd2d7de05be97c0d77feceb7b0" exitCode=60
Oct 30 20:15:18.312998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.312569    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerDied","Data":"1e1740849c8417d9da75d4fb00d1962e2cc4ecdd2d7de05be97c0d77feceb7b0"}
Oct 30 20:15:18.312998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.312646    4917 scope.go:122] "RemoveContainer" containerID="a4973780181595309b266c2018ff0ecefc9deabd7ae9e232a0cf4eec646c61b9"
Oct 30 20:15:18.313142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.313059    4917 scope.go:122] "RemoveContainer" containerID="1e1740849c8417d9da75d4fb00d1962e2cc4ecdd2d7de05be97c0d77feceb7b0"
Oct 30 20:15:18.313488 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:18.313466    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756_odh-ai-gateway-infra(701b2111-b1dd-48d1-bd36-dc379850598c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" podUID="701b2111-b1dd-48d1-bd36-dc379850598c"
Oct 30 20:15:18.319438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.319413    4917 generic.go:356] "Generic (PLEG): container finished" podID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerID="f9bddbfc30f6eb7dff68de49cc169c26b8258fb8ce2e3e734e47fed723d0fbdf" exitCode=60
Oct 30 20:15:18.319562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.319469    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" event={"ID":"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295","Type":"ContainerDied","Data":"f9bddbfc30f6eb7dff68de49cc169c26b8258fb8ce2e3e734e47fed723d0fbdf"}
Oct 30 20:15:18.319898 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.319880    4917 scope.go:122] "RemoveContainer" containerID="f9bddbfc30f6eb7dff68de49cc169c26b8258fb8ce2e3e734e47fed723d0fbdf"
Oct 30 20:15:18.320263 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:18.320231    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8_odh-ai-gateway-infra(e9f6b9e4-b141-4bd4-82fe-8630ee0cf295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295"
Oct 30 20:15:18.326696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.326680    4917 scope.go:122] "RemoveContainer" containerID="8c25e36153a5388d190fd4c2275e5ee442a9bbdb4de929c93f98318f12118028"
Oct 30 20:15:18.424691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.424657    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"]
Oct 30 20:15:18.546612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.546519    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"]
Oct 30 20:15:18.650812 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.642636    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:18.709341 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.709299    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"]
Oct 30 20:15:18.752924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.752891    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n\" (UniqueName: \"kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n\") pod \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") "
Oct 30 20:15:18.753317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.753284    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca\") pod \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\" (UID: \"1ee786e2-f600-4d0a-8853-15f61a5ac87b\") "
Oct 30 20:15:18.753579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.753555    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca" pod "1ee786e2-f600-4d0a-8853-15f61a5ac87b" (UID: "1ee786e2-f600-4d0a-8853-15f61a5ac87b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.753985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.753966    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee786e2-f600-4d0a-8853-15f61a5ac87b-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.755327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.755289    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n" pod "1ee786e2-f600-4d0a-8853-15f61a5ac87b" (UID: "1ee786e2-f600-4d0a-8853-15f61a5ac87b"). InnerVolumeSpecName "kube-api-access-x5v7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.778567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.778537    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:18.780144 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.780123    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"]
Oct 30 20:15:18.854840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.854723    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d\" (UniqueName: \"kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d\") pod \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") "
Oct 30 20:15:18.855010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.854935    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca\") pod \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\" (UID: \"7cd18c01-a9bb-4b82-a531-0734ae14e8ec\") "
Oct 30 20:15:18.855403 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.855371    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca" pod "7cd18c01-a9bb-4b82-a531-0734ae14e8ec" (UID: "7cd18c01-a9bb-4b82-a531-0734ae14e8ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.855760 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.855741    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.855908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.855762    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5v7n\" (UniqueName: \"kubernetes.io/projected/1ee786e2-f600-4d0a-8853-15f61a5ac87b-kube-api-access-x5v7n\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.858288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.858251    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d" pod "7cd18c01-a9bb-4b82-a531-0734ae14e8ec" (UID: "7cd18c01-a9bb-4b82-a531-0734ae14e8ec"). InnerVolumeSpecName "kube-api-access-vsk9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.898367 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.898336    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"]
Oct 30 20:15:18.939624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.939594    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:18.957333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.957296    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsk9d\" (UniqueName: \"kubernetes.io/projected/7cd18c01-a9bb-4b82-a531-0734ae14e8ec-kube-api-access-vsk9d\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.997657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:18.997123    4917 scope.go:122] "RemoveContainer" containerID="fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc"
Oct 30 20:15:19.030690 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.030663    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:19.061806 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.061759    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl\" (UniqueName: \"kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl\") pod \"67d915be-9593-42c8-9a13-4758f10a4e54\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") "
Oct 30 20:15:19.062155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.062134    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca\") pod \"67d915be-9593-42c8-9a13-4758f10a4e54\" (UID: \"67d915be-9593-42c8-9a13-4758f10a4e54\") "
Oct 30 20:15:19.066040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.066009    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca" pod "67d915be-9593-42c8-9a13-4758f10a4e54" (UID: "67d915be-9593-42c8-9a13-4758f10a4e54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.066489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.066461    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl" pod "67d915be-9593-42c8-9a13-4758f10a4e54" (UID: "67d915be-9593-42c8-9a13-4758f10a4e54"). InnerVolumeSpecName "kube-api-access-gkmbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.106837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.106716    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:19.164127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.164096    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca\") pod \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") "
Oct 30 20:15:19.164323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.164307    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c\" (UniqueName: \"kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c\") pod \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\" (UID: \"5e3f9666-894e-4fdc-b378-b4b33ed8d562\") "
Oct 30 20:15:19.164397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.164373    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49\" (UniqueName: \"kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49\") pod \"53236041-b9c6-4c73-a1b5-165288c1923b\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") "
Oct 30 20:15:19.164616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.164492    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca\") pod \"53236041-b9c6-4c73-a1b5-165288c1923b\" (UID: \"53236041-b9c6-4c73-a1b5-165288c1923b\") "
Oct 30 20:15:19.164616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.164556    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca" pod "5e3f9666-894e-4fdc-b378-b4b33ed8d562" (UID: "5e3f9666-894e-4fdc-b378-b4b33ed8d562"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.165026 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.164996    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca" pod "53236041-b9c6-4c73-a1b5-165288c1923b" (UID: "53236041-b9c6-4c73-a1b5-165288c1923b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.165212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.165195    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53236041-b9c6-4c73-a1b5-165288c1923b-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.165280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.165213    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d915be-9593-42c8-9a13-4758f10a4e54-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.165280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.165223    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3f9666-894e-4fdc-b378-b4b33ed8d562-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.165280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.165232    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmbl\" (UniqueName: \"kubernetes.io/projected/67d915be-9593-42c8-9a13-4758f10a4e54-kube-api-access-gkmbl\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.166556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.166531    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49" pod "53236041-b9c6-4c73-a1b5-165288c1923b" (UID: "53236041-b9c6-4c73-a1b5-165288c1923b"). InnerVolumeSpecName "kube-api-access-wnj49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.166649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.166630    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c" pod "5e3f9666-894e-4fdc-b378-b4b33ed8d562" (UID: "5e3f9666-894e-4fdc-b378-b4b33ed8d562"). InnerVolumeSpecName "kube-api-access-fr66c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.266563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.266532    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr66c\" (UniqueName: \"kubernetes.io/projected/5e3f9666-894e-4fdc-b378-b4b33ed8d562-kube-api-access-fr66c\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.266563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.266562    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnj49\" (UniqueName: \"kubernetes.io/projected/53236041-b9c6-4c73-a1b5-165288c1923b-kube-api-access-wnj49\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.337330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.337299    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"
Oct 30 20:15:19.337756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.337297    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r" event={"ID":"53236041-b9c6-4c73-a1b5-165288c1923b","Type":"ContainerDied","Data":"14b7054d245395fd3afd9161eac311fc717b6db2ef7064bb86d54fc1f6b2f462"}
Oct 30 20:15:19.337756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.337493    4917 scope.go:122] "RemoveContainer" containerID="1b3e9024e32c75e5e92fdfb7ec40218a110835376bc293e4c3470f97ecfeb9d1"
Oct 30 20:15:19.339545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.339521    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck" event={"ID":"5e3f9666-894e-4fdc-b378-b4b33ed8d562","Type":"ContainerDied","Data":"8def2c13af78090069b0b6bea8cd5d1c4c1c31ea2cd5a4bfad34aeb4cb205d4a"}
Oct 30 20:15:19.339664 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.339623    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"
Oct 30 20:15:19.350067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.350035    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r" event={"ID":"1ee786e2-f600-4d0a-8853-15f61a5ac87b","Type":"ContainerDied","Data":"ba1745de2ee6020d44e67c4157b2fa7e08fa0e1251abd7d9c7213eb4882c78a8"}
Oct 30 20:15:19.350230 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.350153    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"
Oct 30 20:15:19.354968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.354944    4917 scope.go:122] "RemoveContainer" containerID="8a18e82a93c02123c029b8f63caa6281979492e0ddc3db04d7c33c53d2f24377"
Oct 30 20:15:19.356358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.356332    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv" event={"ID":"67d915be-9593-42c8-9a13-4758f10a4e54","Type":"ContainerDied","Data":"ab97159d6e12a147d6bf3e170b8e89c1261104de41f65fa96adebc2e97e38d3e"}
Oct 30 20:15:19.356484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.356438    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"
Oct 30 20:15:19.364688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.364660    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6" event={"ID":"7cd18c01-a9bb-4b82-a531-0734ae14e8ec","Type":"ContainerDied","Data":"d45bca601515995c09b2679d456dfbcc33a95aeb56be1dc33c4b4b7b84d8e159"}
Oct 30 20:15:19.364849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.364758    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"
Oct 30 20:15:19.385490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.385466    4917 scope.go:122] "RemoveContainer" containerID="6baf908c30aa0f1b0d40cc95e02dad2f8ccf6b231ebf5eecba718909fe394a44"
Oct 30 20:15:19.395905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.395883    4917 scope.go:122] "RemoveContainer" containerID="f734ca68c3842722a6df073e84a184c60360a8bff4f4245f9cf6d00923a60aa1"
Oct 30 20:15:19.405139 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.405117    4917 scope.go:122] "RemoveContainer" containerID="9ccd59ece7f79ff98bf3af3a78db0498c48f219f95ea3d2c67409608a5780d4d"
Oct 30 20:15:19.455623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.455597    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"]
Oct 30 20:15:19.547514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.547481    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"]
Oct 30 20:15:19.560425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.560394    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d67ea9-29889855-5k55r"]
Oct 30 20:15:19.605881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.605856    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:19.669294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.669215    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"]
Oct 30 20:15:19.673558 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.673533    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca\") pod \"701b2111-b1dd-48d1-bd36-dc379850598c\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") "
Oct 30 20:15:19.673695 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.673679    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv\" (UniqueName: \"kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv\") pod \"701b2111-b1dd-48d1-bd36-dc379850598c\" (UID: \"701b2111-b1dd-48d1-bd36-dc379850598c\") "
Oct 30 20:15:19.673907 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.673880    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca" pod "701b2111-b1dd-48d1-bd36-dc379850598c" (UID: "701b2111-b1dd-48d1-bd36-dc379850598c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.674009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.673992    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f75272-29889855-jbbd6"]
Oct 30 20:15:19.674159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.674146    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701b2111-b1dd-48d1-bd36-dc379850598c-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.675733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.675716    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv" pod "701b2111-b1dd-48d1-bd36-dc379850598c" (UID: "701b2111-b1dd-48d1-bd36-dc379850598c"). InnerVolumeSpecName "kube-api-access-q85fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.690436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.690411    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"]
Oct 30 20:15:19.775691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.775659    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85fv\" (UniqueName: \"kubernetes.io/projected/701b2111-b1dd-48d1-bd36-dc379850598c-kube-api-access-q85fv\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.851987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.851955    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"]
Oct 30 20:15:19.856709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.856681    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f44ec2-29889855-qgwqv"]
Oct 30 20:15:19.861636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.861614    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:19.980229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.980192    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg\" (UniqueName: \"kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg\") pod \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") "
Oct 30 20:15:19.980492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.980470    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca\") pod \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\" (UID: \"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295\") "
Oct 30 20:15:19.981048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.981022    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca" pod "e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" (UID: "e9f6b9e4-b141-4bd4-82fe-8630ee0cf295"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.995527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:19.995491    4917 scope.go:122] "RemoveContainer" containerID="d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c"
Oct 30 20:15:20.008516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.008484    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" path="/var/lib/kubelet/pods/1ee786e2-f600-4d0a-8853-15f61a5ac87b/volumes"
Oct 30 20:15:20.009176 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.009153    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" path="/var/lib/kubelet/pods/67d915be-9593-42c8-9a13-4758f10a4e54/volumes"
Oct 30 20:15:20.009752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.009731    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" path="/var/lib/kubelet/pods/7cd18c01-a9bb-4b82-a531-0734ae14e8ec/volumes"
Oct 30 20:15:20.033894 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.033864    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"]
Oct 30 20:15:20.042023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.041488    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg" pod "e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" (UID: "e9f6b9e4-b141-4bd4-82fe-8630ee0cf295"). InnerVolumeSpecName "kube-api-access-f2zgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.047928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.047904    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a2103a-29889855-hk42r"]
Oct 30 20:15:20.083025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.082945    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.083025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.082974    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zgg\" (UniqueName: \"kubernetes.io/projected/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295-kube-api-access-f2zgg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.222446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.222411    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"]
Oct 30 20:15:20.227202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.227177    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889855-92mck"]
Oct 30 20:15:20.385641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.385610    4917 generic.go:356] "Generic (PLEG): container finished" podID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerID="d46bb73a5fad0ca6eac9b93d46619df30d9ca1b201edc451f1261e7b213343d9" exitCode=60
Oct 30 20:15:20.386159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.385692    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" event={"ID":"d8dbd364-a935-4fac-8d75-71a6e93e7f58","Type":"ContainerDied","Data":"d46bb73a5fad0ca6eac9b93d46619df30d9ca1b201edc451f1261e7b213343d9"}
Oct 30 20:15:20.386159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.385842    4917 scope.go:122] "RemoveContainer" containerID="fc8f47a25fc407bb5cb5af218bb47c31b23e434dae74c48875b97d0ab1b329dc"
Oct 30 20:15:20.386305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.386284    4917 scope.go:122] "RemoveContainer" containerID="d46bb73a5fad0ca6eac9b93d46619df30d9ca1b201edc451f1261e7b213343d9"
Oct 30 20:15:20.386711 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:20.386684    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb_odh-ai-gateway-infra(d8dbd364-a935-4fac-8d75-71a6e93e7f58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58"
Oct 30 20:15:20.387497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.387395    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8" event={"ID":"e9f6b9e4-b141-4bd4-82fe-8630ee0cf295","Type":"ContainerDied","Data":"06e1e4d2803beb5759d9fbaf50fbf6d5a5ac4be6512bac830e86067a1d34829a"}
Oct 30 20:15:20.387497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.387447    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"
Oct 30 20:15:20.392318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.391985    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"
Oct 30 20:15:20.392318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.391983    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756" event={"ID":"701b2111-b1dd-48d1-bd36-dc379850598c","Type":"ContainerDied","Data":"0ed5f974e22bc2e3a0a35efa197bb99fbaa012d951f5735a9a7bf85731e4bbd2"}
Oct 30 20:15:20.405038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.405018    4917 scope.go:122] "RemoveContainer" containerID="f9bddbfc30f6eb7dff68de49cc169c26b8258fb8ce2e3e734e47fed723d0fbdf"
Oct 30 20:15:20.419402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.419373    4917 scope.go:122] "RemoveContainer" containerID="1e1740849c8417d9da75d4fb00d1962e2cc4ecdd2d7de05be97c0d77feceb7b0"
Oct 30 20:15:20.580518 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.580437    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"]
Oct 30 20:15:20.584567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.584541    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7961ed-29889855-ft756"]
Oct 30 20:15:20.699908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.699872    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"]
Oct 30 20:15:20.704216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:20.704187    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889855-8q8g8"]
Oct 30 20:15:21.410051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:21.409974    4917 generic.go:356] "Generic (PLEG): container finished" podID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerID="722d91d3562761efa4ce531299e200d00a7b9c66db4500053c0ed2ce2cac4a65" exitCode=60
Oct 30 20:15:21.410485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:21.410045    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerDied","Data":"722d91d3562761efa4ce531299e200d00a7b9c66db4500053c0ed2ce2cac4a65"}
Oct 30 20:15:21.410485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:21.410102    4917 scope.go:122] "RemoveContainer" containerID="d3aa90f370343e5376e366ebe2d4f43bd7db6870b4778dbf7a9f9942c1ecd17c"
Oct 30 20:15:21.410555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:21.410511    4917 scope.go:122] "RemoveContainer" containerID="722d91d3562761efa4ce531299e200d00a7b9c66db4500053c0ed2ce2cac4a65"
Oct 30 20:15:21.410890 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:21.410868    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl_odh-ai-gateway-infra(21c35eac-3545-4cd4-a171-8a75585aeaa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4"
Oct 30 20:15:21.839424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:21.839391    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"]
Oct 30 20:15:22.008923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.008888    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" path="/var/lib/kubelet/pods/53236041-b9c6-4c73-a1b5-165288c1923b/volumes"
Oct 30 20:15:22.009654 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.009628    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" path="/var/lib/kubelet/pods/5e3f9666-894e-4fdc-b378-b4b33ed8d562/volumes"
Oct 30 20:15:22.010298 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.010274    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" path="/var/lib/kubelet/pods/701b2111-b1dd-48d1-bd36-dc379850598c/volumes"
Oct 30 20:15:22.022555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.011382    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" path="/var/lib/kubelet/pods/e9f6b9e4-b141-4bd4-82fe-8630ee0cf295/volumes"
Oct 30 20:15:22.029993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.029972    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:22.115060 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.114975    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655\" (UniqueName: \"kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655\") pod \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") "
Oct 30 20:15:22.115208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.115156    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca\") pod \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\" (UID: \"d8dbd364-a935-4fac-8d75-71a6e93e7f58\") "
Oct 30 20:15:22.115500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.115472    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca" pod "d8dbd364-a935-4fac-8d75-71a6e93e7f58" (UID: "d8dbd364-a935-4fac-8d75-71a6e93e7f58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:22.115730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.115717    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbd364-a935-4fac-8d75-71a6e93e7f58-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.117251 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.117228    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655" pod "d8dbd364-a935-4fac-8d75-71a6e93e7f58" (UID: "d8dbd364-a935-4fac-8d75-71a6e93e7f58"). InnerVolumeSpecName "kube-api-access-b8655". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:22.217230 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.217199    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8655\" (UniqueName: \"kubernetes.io/projected/d8dbd364-a935-4fac-8d75-71a6e93e7f58-kube-api-access-b8655\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.425718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.425634    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"
Oct 30 20:15:22.425718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.425645    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb" event={"ID":"d8dbd364-a935-4fac-8d75-71a6e93e7f58","Type":"ContainerDied","Data":"0b22648953181dc7eb834e7b990b91e491fda877b1bd8e660112f9fee4fea1cb"}
Oct 30 20:15:22.426247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.425724    4917 scope.go:122] "RemoveContainer" containerID="d46bb73a5fad0ca6eac9b93d46619df30d9ca1b201edc451f1261e7b213343d9"
Oct 30 20:15:22.540453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.540420    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"]
Oct 30 20:15:22.623792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.620639    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"]
Oct 30 20:15:22.638242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.638206    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-577fb85a-29889855-hlssb"]
Oct 30 20:15:22.802952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.802922    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:22.933890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.933858    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb\" (UniqueName: \"kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb\") pod \"21c35eac-3545-4cd4-a171-8a75585aeaa4\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") "
Oct 30 20:15:22.934055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.933921    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca\") pod \"21c35eac-3545-4cd4-a171-8a75585aeaa4\" (UID: \"21c35eac-3545-4cd4-a171-8a75585aeaa4\") "
Oct 30 20:15:22.934391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.934362    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca" pod "21c35eac-3545-4cd4-a171-8a75585aeaa4" (UID: "21c35eac-3545-4cd4-a171-8a75585aeaa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:22.934610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.934585    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c35eac-3545-4cd4-a171-8a75585aeaa4-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.936316 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:22.936294    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb" pod "21c35eac-3545-4cd4-a171-8a75585aeaa4" (UID: "21c35eac-3545-4cd4-a171-8a75585aeaa4"). InnerVolumeSpecName "kube-api-access-jl9fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:23.036301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.036222    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl9fb\" (UniqueName: \"kubernetes.io/projected/21c35eac-3545-4cd4-a171-8a75585aeaa4-kube-api-access-jl9fb\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.434448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.434417    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"
Oct 30 20:15:23.434944 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.434444    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl" event={"ID":"21c35eac-3545-4cd4-a171-8a75585aeaa4","Type":"ContainerDied","Data":"eb5a7be7fdfc98775f195a42726dbfe6cb14c3a782ad948f1f3156e227160b8c"}
Oct 30 20:15:23.434944 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.434525    4917 scope.go:122] "RemoveContainer" containerID="722d91d3562761efa4ce531299e200d00a7b9c66db4500053c0ed2ce2cac4a65"
Oct 30 20:15:23.600019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.599986    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:23.604329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.604301    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"]
Oct 30 20:15:23.613029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.613006    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:23.653864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:23.653832    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889855-r4czl"]
Oct 30 20:15:24.013953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:24.013912    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" path="/var/lib/kubelet/pods/21c35eac-3545-4cd4-a171-8a75585aeaa4/volumes"
Oct 30 20:15:24.014548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:24.014523    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" path="/var/lib/kubelet/pods/d8dbd364-a935-4fac-8d75-71a6e93e7f58/volumes"
Oct 30 20:15:24.456407 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:24.456378    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:15:39.682723 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:15:39.682679    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:15:44.978005 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:44.977972    4917 scope.go:122] "RemoveContainer" containerID="4dfc362c48953ac624441ae73788321b5d42a79bc4cb97a10be40a440bbdf7e4"
Oct 30 20:15:44.990218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:44.990193    4917 scope.go:122] "RemoveContainer" containerID="436f884326ecfefdae510a44fd00d24cdc54cee6970ceb026d75924808b13126"
Oct 30 20:15:45.002325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:45.002299    4917 scope.go:122] "RemoveContainer" containerID="2337d092c37842f0ae99733c6e8469d070504ba0e7e530446fbf5d91e2225c9d"
Oct 30 20:15:45.013363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:45.013339    4917 scope.go:122] "RemoveContainer" containerID="ec96acd74a746e26a3baee9a09779df948148f880e17540546861d0961a81293"
Oct 30 20:15:45.022453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:45.022433    4917 scope.go:122] "RemoveContainer" containerID="351f640aeeb424d9495891dd0060bc97c69d5f3cc6b6c2605125e17163ad116b"
Oct 30 20:15:45.049552 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:15:45.049527    4917 scope.go:122] "RemoveContainer" containerID="5903c1a5baeaa7af89e6b58db4ee803b7eca7f9ee8fc176046ba8fa774fae6b5"
Oct 30 20:16:09.680375 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:16:09.680336    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:16:13.057384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:13.057348    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"]
Oct 30 20:16:13.106829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:13.106786    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcf6c4eb-6mccn"]
Oct 30 20:16:14.004648 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:14.004619    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b94266-b4f6-4199-96ce-c6bca2396bd5" path="/var/lib/kubelet/pods/a8b94266-b4f6-4199-96ce-c6bca2396bd5/volumes"
Oct 30 20:16:18.892828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:18.886589    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"]
Oct 30 20:16:18.895377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:18.895347    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577fb85a-55fxd"]
Oct 30 20:16:18.901942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:18.901911    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:16:20.008141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:20.008107    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f692ddd8-19ce-445a-b202-0a5825349763" path="/var/lib/kubelet/pods/f692ddd8-19ce-445a-b202-0a5825349763/volumes"
Oct 30 20:16:30.262940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.262899    4917 patch_prober.go:29] interesting pod/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:16:30.263391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.262970    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:16:30.722084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.722054    4917 generic.go:356] "Generic (PLEG): container finished" podID="795942c1-11a0-4218-a931-a6a0807d8adf" containerID="6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4" exitCode=0
Oct 30 20:16:30.722264 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.722129    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" event={"ID":"795942c1-11a0-4218-a931-a6a0807d8adf","Type":"ContainerDied","Data":"6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"}
Oct 30 20:16:30.722710 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.722691    4917 scope.go:122] "RemoveContainer" containerID="6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"
Oct 30 20:16:30.724620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.724587    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:16:30.724747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.724694    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:16:30.724832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:30.724746    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:16:31.732535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:31.732499    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" event={"ID":"795942c1-11a0-4218-a931-a6a0807d8adf","Type":"ContainerStarted","Data":"eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6"}
Oct 30 20:16:32.227121 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:32.227084    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:16:32.227303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:32.227276    4917 patch_prober.go:29] interesting pod/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:16:32.227351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:32.227323    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:16:33.227199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:33.227162    4917 patch_prober.go:29] interesting pod/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:16:33.227600 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:33.227247    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:16:34.227301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:34.227255    4917 patch_prober.go:29] interesting pod/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:16:34.227729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:34.227350    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:16:35.267797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:35.252591    4917 patch_prober.go:29] interesting pod/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:16:35.268471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:35.268432    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:16:36.227309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:36.227269    4917 patch_prober.go:29] interesting pod/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:16:36.227494 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:36.227338    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 20:16:37.231840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:37.231762    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:16:37.232329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:37.232292    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:16:37.233069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:37.233046    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:16:39.681918 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:16:39.681880    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:16:44.290625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:44.290589    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:16:44.337016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:44.336989    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:16:57.508264 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.508216    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"]
Oct 30 20:16:57.509236 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509212    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.509293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509241    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.509293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509258    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:16:57.509293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509267    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:16:57.509293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509284    4917 cpu_manager.go:414] "containerMap: removing container" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509293    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509309    4917 cpu_manager.go:414] "containerMap: removing container" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509315    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509334    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509341    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509352    4917 cpu_manager.go:414] "containerMap: removing container" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509359    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509370    4917 cpu_manager.go:414] "containerMap: removing container" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509379    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509402    4917 cpu_manager.go:414] "containerMap: removing container" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509410    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509430    4917 cpu_manager.go:414] "containerMap: removing container" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.509454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509438    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509461    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509468    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509477    4917 cpu_manager.go:414] "containerMap: removing container" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509485    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509497    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509505    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509516    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509523    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509532    4917 cpu_manager.go:414] "containerMap: removing container" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509540    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509559    4917 cpu_manager.go:414] "containerMap: removing container" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509567    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509582    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509589    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509608    4917 cpu_manager.go:414] "containerMap: removing container" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509614    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509624    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509632    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509642    4917 cpu_manager.go:414] "containerMap: removing container" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509648    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509657    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509666    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509685    4917 cpu_manager.go:414] "containerMap: removing container" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509693    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509890    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509907    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509916    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509927    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509934    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509947    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509958    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509967    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.509976    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:16:57.509983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510007    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510019    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510030    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="53236041-b9c6-4c73-a1b5-165288c1923b" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510039    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510048    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510063    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510072    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510079    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510087    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510095    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510104    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.511098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.510116    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:16:57.513439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.513418    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.570292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.570254    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"]
Oct 30 20:16:57.597474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.597416    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbd94\" (UniqueName: \"kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.597661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.597511    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.597661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.597603    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.614216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.614177    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:16:57.614372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.614238    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:16:57.614492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.614199    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:57.616929 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.616899    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:16:57.699148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.699117    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbd94\" (UniqueName: \"kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.699320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.699173    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.699320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.699203    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.699903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.699885    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.701712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.701687    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.761463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.761404    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbd94\" (UniqueName: \"kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:57.828050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:57.828018    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:16:58.211121 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:58.211092    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"]
Oct 30 20:16:58.212163 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:16:58.212135    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36ce220_5773_4884_b5c8_7913c443fdaa.slice/crio-b178cb678ba178d705614aaf785764b36612e4b186f25fd15467b822f71dee8e WatchSource:0}: Error finding container b178cb678ba178d705614aaf785764b36612e4b186f25fd15467b822f71dee8e: Status 404 returned error can't find the container with id b178cb678ba178d705614aaf785764b36612e4b186f25fd15467b822f71dee8e
Oct 30 20:16:58.626924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:58.626882    4917 patch_prober.go:29] interesting pod/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:16:58.627336 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:58.626948    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:16:59.074243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:59.074208    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr" event={"ID":"b36ce220-5773-4884-b5c8-7913c443fdaa","Type":"ContainerStarted","Data":"2df74d4cbe7bbeef1a00697ef8f94b4cbd93d73b349be494d06bc96f053b5b73"}
Oct 30 20:16:59.074430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:59.074270    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr" event={"ID":"b36ce220-5773-4884-b5c8-7913c443fdaa","Type":"ContainerStarted","Data":"b178cb678ba178d705614aaf785764b36612e4b186f25fd15467b822f71dee8e"}
Oct 30 20:16:59.196900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:16:59.196832    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr" podStartSLOduration=2.19681202 podStartE2EDuration="2.19681202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:59.191719047 +0000 UTC m=+2115.356120689" watchObservedRunningTime="2026-10-30 20:16:59.19681202 +0000 UTC m=+2115.361213664"
Oct 30 20:17:00.083459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.083425    4917 generic.go:356] "Generic (PLEG): container finished" podID="b36ce220-5773-4884-b5c8-7913c443fdaa" containerID="2df74d4cbe7bbeef1a00697ef8f94b4cbd93d73b349be494d06bc96f053b5b73" exitCode=0
Oct 30 20:17:00.084002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.083511    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr" event={"ID":"b36ce220-5773-4884-b5c8-7913c443fdaa","Type":"ContainerDied","Data":"2df74d4cbe7bbeef1a00697ef8f94b4cbd93d73b349be494d06bc96f053b5b73"}
Oct 30 20:17:00.085320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.085298    4917 generic.go:356] "Generic (PLEG): container finished" podID="93420d97-cb7f-4516-b7bd-75922905beb3" containerID="9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95" exitCode=0
Oct 30 20:17:00.085428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.085376    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" event={"ID":"93420d97-cb7f-4516-b7bd-75922905beb3","Type":"ContainerDied","Data":"9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"}
Oct 30 20:17:00.085839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.085822    4917 scope.go:122] "RemoveContainer" containerID="9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"
Oct 30 20:17:00.087512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.087484    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:00.087587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.087555    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:00.087630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:00.087589    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:17:01.109884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.109845    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" event={"ID":"93420d97-cb7f-4516-b7bd-75922905beb3","Type":"ContainerStarted","Data":"0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934"}
Oct 30 20:17:01.324062 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.324036    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:17:01.477822 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.477617    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca\") pod \"b36ce220-5773-4884-b5c8-7913c443fdaa\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") "
Oct 30 20:17:01.478094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.478059    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94\" (UniqueName: \"kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94\") pod \"b36ce220-5773-4884-b5c8-7913c443fdaa\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") "
Oct 30 20:17:01.478094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.478069    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca" pod "b36ce220-5773-4884-b5c8-7913c443fdaa" (UID: "b36ce220-5773-4884-b5c8-7913c443fdaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:01.478269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.478227    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca\") pod \"b36ce220-5773-4884-b5c8-7913c443fdaa\" (UID: \"b36ce220-5773-4884-b5c8-7913c443fdaa\") "
Oct 30 20:17:01.478965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.478915    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:01.480741 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.480714    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca" pod "b36ce220-5773-4884-b5c8-7913c443fdaa" (UID: "b36ce220-5773-4884-b5c8-7913c443fdaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:01.481238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.481210    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94" pod "b36ce220-5773-4884-b5c8-7913c443fdaa" (UID: "b36ce220-5773-4884-b5c8-7913c443fdaa"). InnerVolumeSpecName "kube-api-access-qbd94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:01.580966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.580928    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbd94\" (UniqueName: \"kubernetes.io/projected/b36ce220-5773-4884-b5c8-7913c443fdaa-kube-api-access-qbd94\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:01.580966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.580960    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b36ce220-5773-4884-b5c8-7913c443fdaa-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:01.938926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.938880    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:17:01.939222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.939193    4917 patch_prober.go:29] interesting pod/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:17:01.939297 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:01.939243    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 20:17:02.119366 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:02.119337    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"
Oct 30 20:17:02.119833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:02.119371    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr" event={"ID":"b36ce220-5773-4884-b5c8-7913c443fdaa","Type":"ContainerDied","Data":"b178cb678ba178d705614aaf785764b36612e4b186f25fd15467b822f71dee8e"}
Oct 30 20:17:02.119833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:02.119427    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b178cb678ba178d705614aaf785764b36612e4b186f25fd15467b822f71dee8e"
Oct 30 20:17:02.939332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:02.939299    4917 patch_prober.go:29] interesting pod/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:17:02.939504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:02.939350    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 20:17:03.939706 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:03.939674    4917 patch_prober.go:29] interesting pod/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:17:03.940156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:03.939722    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 20:17:04.939027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:04.938988    4917 patch_prober.go:29] interesting pod/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:17:04.939208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:04.939042    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 20:17:05.938898 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:05.938859    4917 patch_prober.go:29] interesting pod/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:17:05.939406 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:05.938922    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 20:17:06.943980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:06.943947    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:17:06.944407 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:06.944388    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:17:06.945008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:06.944988    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:17:09.708549 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:09.708503    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:17:12.100759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.100718    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"]
Oct 30 20:17:12.101567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101542    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101568    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101591    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101598    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee786e2-f600-4d0a-8853-15f61a5ac87b" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101609    4917 cpu_manager.go:414] "containerMap: removing container" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101617    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101649    4917 cpu_manager.go:414] "containerMap: removing container" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:17:12.101686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101657    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c35eac-3545-4cd4-a171-8a75585aeaa4" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101703    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b36ce220-5773-4884-b5c8-7913c443fdaa" containerName="revoke-keys"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101712    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36ce220-5773-4884-b5c8-7913c443fdaa" containerName="revoke-keys"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101741    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101749    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101784    4917 cpu_manager.go:414] "containerMap: removing container" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101792    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101975    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="701b2111-b1dd-48d1-bd36-dc379850598c" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.101995    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36ce220-5773-4884-b5c8-7913c443fdaa" containerName="revoke-keys"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.102005    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d915be-9593-42c8-9a13-4758f10a4e54" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.102017    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dbd364-a935-4fac-8d75-71a6e93e7f58" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.102030    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3f9666-894e-4fdc-b378-b4b33ed8d562" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.102040    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f6b9e4-b141-4bd4-82fe-8630ee0cf295" containerName="cleanup"
Oct 30 20:17:12.102094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.102051    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd18c01-a9bb-4b82-a531-0734ae14e8ec" containerName="cleanup"
Oct 30 20:17:12.105114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.105092    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.164641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.164595    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"]
Oct 30 20:17:12.199550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.199501    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.199884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.199860    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfksm\" (UniqueName: \"kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.199965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.199928    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.209176 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.209147    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:12.209365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.209158    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:17:12.209459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.209164    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:17:12.209459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.209227    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:17:12.301307 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.301274    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.301476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.301463    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfksm\" (UniqueName: \"kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.301521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.301505    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.302127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.302101    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.303760 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.303728    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.363379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.363314    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfksm\" (UniqueName: \"kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm\") pod \"maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.417683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.417652    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:12.803722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:12.803691    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"]
Oct 30 20:17:12.803891 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:17:12.803868    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b401f60_4e13_47fe_afd7_1ec533a659fd.slice/crio-fb32ef8940a1da986a6843f7c919f211a87b01da8f40b35598b9e6fd574f2c7f WatchSource:0}: Error finding container fb32ef8940a1da986a6843f7c919f211a87b01da8f40b35598b9e6fd574f2c7f: Status 404 returned error can't find the container with id fb32ef8940a1da986a6843f7c919f211a87b01da8f40b35598b9e6fd574f2c7f
Oct 30 20:17:13.226446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:13.226395    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx" event={"ID":"4b401f60-4e13-47fe-afd7-1ec533a659fd","Type":"ContainerStarted","Data":"1af0e18194425b3b2f3291814220d76de20247b96492459543302e3d28548ddc"}
Oct 30 20:17:13.226851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:13.226484    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx" event={"ID":"4b401f60-4e13-47fe-afd7-1ec533a659fd","Type":"ContainerStarted","Data":"fb32ef8940a1da986a6843f7c919f211a87b01da8f40b35598b9e6fd574f2c7f"}
Oct 30 20:17:13.344831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:13.344713    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx" podStartSLOduration=1.344698505 podStartE2EDuration="1.344698505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:13.343289759 +0000 UTC m=+2129.507691403" watchObservedRunningTime="2026-10-30 20:17:13.344698505 +0000 UTC m=+2129.509100189"
Oct 30 20:17:14.235569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:14.235537    4917 generic.go:356] "Generic (PLEG): container finished" podID="4b401f60-4e13-47fe-afd7-1ec533a659fd" containerID="1af0e18194425b3b2f3291814220d76de20247b96492459543302e3d28548ddc" exitCode=0
Oct 30 20:17:14.236085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:14.235634    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx" event={"ID":"4b401f60-4e13-47fe-afd7-1ec533a659fd","Type":"ContainerDied","Data":"1af0e18194425b3b2f3291814220d76de20247b96492459543302e3d28548ddc"}
Oct 30 20:17:15.207160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.207122    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"]
Oct 30 20:17:15.210392 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.210367    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.247551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.247520    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xh7\" (UniqueName: \"kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.247982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.247732    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.247982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.247922    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.269820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.269761    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"]
Oct 30 20:17:15.350075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.350041    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.350245 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.350163    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.350371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.350353    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xh7\" (UniqueName: \"kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.351472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.351445    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.353682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.353660    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.413501 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.413458    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xh7\" (UniqueName: \"kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7\") pod \"maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.446547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.446517    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:15.523613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.523542    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:15.552765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.552729    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm\" (UniqueName: \"kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm\") pod \"4b401f60-4e13-47fe-afd7-1ec533a659fd\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") "
Oct 30 20:17:15.552953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.552926    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca\") pod \"4b401f60-4e13-47fe-afd7-1ec533a659fd\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") "
Oct 30 20:17:15.553051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.553031    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca\") pod \"4b401f60-4e13-47fe-afd7-1ec533a659fd\" (UID: \"4b401f60-4e13-47fe-afd7-1ec533a659fd\") "
Oct 30 20:17:15.553934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.553904    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca" pod "4b401f60-4e13-47fe-afd7-1ec533a659fd" (UID: "4b401f60-4e13-47fe-afd7-1ec533a659fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:15.555363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.555333    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca" pod "4b401f60-4e13-47fe-afd7-1ec533a659fd" (UID: "4b401f60-4e13-47fe-afd7-1ec533a659fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:15.555363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.555353    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm" pod "4b401f60-4e13-47fe-afd7-1ec533a659fd" (UID: "4b401f60-4e13-47fe-afd7-1ec533a659fd"). InnerVolumeSpecName "kube-api-access-mfksm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:15.655730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.655694    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:15.655730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.655729    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfksm\" (UniqueName: \"kubernetes.io/projected/4b401f60-4e13-47fe-afd7-1ec533a659fd-kube-api-access-mfksm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:15.656005 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.655749    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b401f60-4e13-47fe-afd7-1ec533a659fd-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:15.908197 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:17:15.908165    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dbc411_d8a6_4aa4_9bbd_d5ebed916379.slice/crio-773d2ca3f407a86b4b9e407454112d12d378cd34c5a1fa419889956a871b93ba WatchSource:0}: Error finding container 773d2ca3f407a86b4b9e407454112d12d378cd34c5a1fa419889956a871b93ba: Status 404 returned error can't find the container with id 773d2ca3f407a86b4b9e407454112d12d378cd34c5a1fa419889956a871b93ba
Oct 30 20:17:15.909984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:15.909962    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"]
Oct 30 20:17:16.253113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:16.253086    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"
Oct 30 20:17:16.253561 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:16.253091    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx" event={"ID":"4b401f60-4e13-47fe-afd7-1ec533a659fd","Type":"ContainerDied","Data":"fb32ef8940a1da986a6843f7c919f211a87b01da8f40b35598b9e6fd574f2c7f"}
Oct 30 20:17:16.253561 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:16.253205    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb32ef8940a1da986a6843f7c919f211a87b01da8f40b35598b9e6fd574f2c7f"
Oct 30 20:17:16.255236 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:16.255205    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq" event={"ID":"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379","Type":"ContainerStarted","Data":"773d2ca3f407a86b4b9e407454112d12d378cd34c5a1fa419889956a871b93ba"}
Oct 30 20:17:17.265638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:17.265598    4917 generic.go:356] "Generic (PLEG): container finished" podID="f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" containerID="419fca8092562d58d5db9b8384a19ba1a8747d09845aeaa7eab7e9a54ae33169" exitCode=0
Oct 30 20:17:17.266123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:17.265675    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq" event={"ID":"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379","Type":"ContainerDied","Data":"419fca8092562d58d5db9b8384a19ba1a8747d09845aeaa7eab7e9a54ae33169"}
Oct 30 20:17:18.452110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.452082    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:18.599193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.599119    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7\" (UniqueName: \"kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7\") pod \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") "
Oct 30 20:17:18.599351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.599301    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca\") pod \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") "
Oct 30 20:17:18.599351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.599343    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca\") pod \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\" (UID: \"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379\") "
Oct 30 20:17:18.599832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.599800    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca" pod "f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" (UID: "f9dbc411-d8a6-4aa4-9bbd-d5ebed916379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:18.600013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.599994    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:18.602056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.602027    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca" pod "f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" (UID: "f9dbc411-d8a6-4aa4-9bbd-d5ebed916379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:18.602178 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.602034    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7" pod "f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" (UID: "f9dbc411-d8a6-4aa4-9bbd-d5ebed916379"). InnerVolumeSpecName "kube-api-access-c5xh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:18.701116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.701081    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:18.701116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:18.701112    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5xh7\" (UniqueName: \"kubernetes.io/projected/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379-kube-api-access-c5xh7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:19.287617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:19.287589    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"
Oct 30 20:17:19.287833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:19.287631    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq" event={"ID":"f9dbc411-d8a6-4aa4-9bbd-d5ebed916379","Type":"ContainerDied","Data":"773d2ca3f407a86b4b9e407454112d12d378cd34c5a1fa419889956a871b93ba"}
Oct 30 20:17:19.287833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:19.287681    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773d2ca3f407a86b4b9e407454112d12d378cd34c5a1fa419889956a871b93ba"
Oct 30 20:17:28.216196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:28.216161    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:17:33.061946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:33.061912    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"]
Oct 30 20:17:33.112907 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:33.112868    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8a1a65-fgghk"]
Oct 30 20:17:34.004840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:34.004806    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3fd2c0-80e6-4000-9f87-d7698cac02c3" path="/var/lib/kubelet/pods/0a3fd2c0-80e6-4000-9f87-d7698cac02c3/volumes"
Oct 30 20:17:37.061512 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:37.061474    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"]
Oct 30 20:17:37.062024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:37.061682    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" containerID="cri-o://d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89" gracePeriod=30
Oct 30 20:17:39.680383 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:39.680345    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:17:40.600694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:40.600655    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:40.600928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:40.600728    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:42.359301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.359275    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:17:42.418094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418067    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418127    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418297 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418258    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418344    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418346    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:42.418410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418382    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418466    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418518    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:42.418625 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418614    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418633    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:42.418680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418656    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418860 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418843    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo\") pod \"408c02f5-9912-4a99-b957-b07c20128df0\" (UID: \"408c02f5-9912-4a99-b957-b07c20128df0\") "
Oct 30 20:17:42.418943 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.418894    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:42.419085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.419050    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:42.419390 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.419373    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.419463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.419398    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.419463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.419413    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.419463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.419427    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.419463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.419440    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408c02f5-9912-4a99-b957-b07c20128df0-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.420569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.420543    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "kube-api-access-vjrsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:42.420569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.420557    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:42.420994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.420975    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:42.421082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.421067    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo" pod "408c02f5-9912-4a99-b957-b07c20128df0" (UID: "408c02f5-9912-4a99-b957-b07c20128df0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:42.521371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.521336    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408c02f5-9912-4a99-b957-b07c20128df0-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.521371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.521367    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.521551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.521381    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrsw\" (UniqueName: \"kubernetes.io/projected/408c02f5-9912-4a99-b957-b07c20128df0-kube-api-access-vjrsw\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.521551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.521396    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408c02f5-9912-4a99-b957-b07c20128df0-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.560869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.560833    4917 generic.go:356] "Generic (PLEG): container finished" podID="408c02f5-9912-4a99-b957-b07c20128df0" containerID="d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89" exitCode=0
Oct 30 20:17:42.561109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.560903    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"
Oct 30 20:17:42.561109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.560978    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" event={"ID":"408c02f5-9912-4a99-b957-b07c20128df0","Type":"ContainerDied","Data":"d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89"}
Oct 30 20:17:42.561109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.561021    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2" event={"ID":"408c02f5-9912-4a99-b957-b07c20128df0","Type":"ContainerDied","Data":"a51f2cc9d0ca55b9506a950347c69064cf799f632f8f8d8aedad93ee30706e93"}
Oct 30 20:17:42.561109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.561049    4917 scope.go:122] "RemoveContainer" containerID="d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89"
Oct 30 20:17:42.575211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.575157    4917 scope.go:122] "RemoveContainer" containerID="d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89"
Oct 30 20:17:42.575620 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:42.575592    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89\": container with ID starting with d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89 not found: ID does not exist" containerID="d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89"
Oct 30 20:17:42.575703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.575632    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89"} err="failed to get container status \"d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89\": rpc error: code = NotFound desc = could not find container \"d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89\": container with ID starting with d30d8c8c7a9e2e3b12d525e28e88f2f3e3b747e67bd58b10867d234ebff0ee89 not found: ID does not exist"
Oct 30 20:17:42.738882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.738841    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"]
Oct 30 20:17:42.744834 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:42.744808    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-qkvw2"]
Oct 30 20:17:44.003411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.003384    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408c02f5-9912-4a99-b957-b07c20128df0" path="/var/lib/kubelet/pods/408c02f5-9912-4a99-b957-b07c20128df0/volumes"
Oct 30 20:17:44.301590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.301520    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"]
Oct 30 20:17:44.302155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302140    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4b401f60-4e13-47fe-afd7-1ec533a659fd" containerName="revoke-keys"
Oct 30 20:17:44.302212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302157    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b401f60-4e13-47fe-afd7-1ec533a659fd" containerName="revoke-keys"
Oct 30 20:17:44.302212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302179    4917 cpu_manager.go:414] "containerMap: removing container" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy"
Oct 30 20:17:44.302212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302184    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy"
Oct 30 20:17:44.302212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302194    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" containerName="revoke-keys"
Oct 30 20:17:44.302212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302199    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" containerName="revoke-keys"
Oct 30 20:17:44.302378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302291    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b401f60-4e13-47fe-afd7-1ec533a659fd" containerName="revoke-keys"
Oct 30 20:17:44.302378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302304    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="408c02f5-9912-4a99-b957-b07c20128df0" containerName="istio-proxy"
Oct 30 20:17:44.302378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.302310    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" containerName="revoke-keys"
Oct 30 20:17:44.307195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.307174    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.345024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.344998    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.345135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.345124    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpxw\" (UniqueName: \"kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.345224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.345211    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.367222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.367188    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"]
Oct 30 20:17:44.409683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.409643    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:44.409683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.409679    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:17:44.409920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.409678    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:17:44.410872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.410852    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:17:44.446326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.446298    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpxw\" (UniqueName: \"kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.446473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.446371    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.446537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.446512    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.447184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.447166    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.448807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.448765    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.508879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.508854    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpxw\" (UniqueName: \"kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-gglnq\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:44.620551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:44.620466    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:45.003009 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:17:45.002970    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87163a46_7b84_4f72_bbf5_7335d1715d5c.slice/crio-9254ed20023c6014e9aaa2a3bcd420e754ed86e8fd24d901876c6aeead8defe0 WatchSource:0}: Error finding container 9254ed20023c6014e9aaa2a3bcd420e754ed86e8fd24d901876c6aeead8defe0: Status 404 returned error can't find the container with id 9254ed20023c6014e9aaa2a3bcd420e754ed86e8fd24d901876c6aeead8defe0
Oct 30 20:17:45.003156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.003066    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"]
Oct 30 20:17:45.170589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.170546    4917 scope.go:122] "RemoveContainer" containerID="f9833c61cbcbf2b37a1e436511499f5bc05138a87286831d3f4d81775ac40459"
Oct 30 20:17:45.180882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.180860    4917 scope.go:122] "RemoveContainer" containerID="69cb70e4dbed48df91f78c0f6010eaf8611b527296b91ca29dc912e0a0959101"
Oct 30 20:17:45.190205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.190183    4917 scope.go:122] "RemoveContainer" containerID="e1f0eee70f93041a01f3e02f27391265ddb662750aa886efaff8739767ce9f94"
Oct 30 20:17:45.200318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.200295    4917 scope.go:122] "RemoveContainer" containerID="d346c2c11204cdc0ed4c2815e338a15014fb9560f878d964cee39c6220af1dd0"
Oct 30 20:17:45.586132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.586039    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq" event={"ID":"87163a46-7b84-4f72-bbf5-7335d1715d5c","Type":"ContainerStarted","Data":"537aebc6900c1658c5eaecfe10d9cb78b336c1fd4f4efbd3c1990aace5340a2f"}
Oct 30 20:17:45.586132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.586102    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq" event={"ID":"87163a46-7b84-4f72-bbf5-7335d1715d5c","Type":"ContainerStarted","Data":"9254ed20023c6014e9aaa2a3bcd420e754ed86e8fd24d901876c6aeead8defe0"}
Oct 30 20:17:45.707475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:45.707423    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq" podStartSLOduration=1.707409609 podStartE2EDuration="1.707409609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:45.704280179 +0000 UTC m=+2161.868681821" watchObservedRunningTime="2026-10-30 20:17:45.707409609 +0000 UTC m=+2161.871811250"
Oct 30 20:17:46.595085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:46.595053    4917 generic.go:356] "Generic (PLEG): container finished" podID="87163a46-7b84-4f72-bbf5-7335d1715d5c" containerID="537aebc6900c1658c5eaecfe10d9cb78b336c1fd4f4efbd3c1990aace5340a2f" exitCode=0
Oct 30 20:17:46.595633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:46.595137    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq" event={"ID":"87163a46-7b84-4f72-bbf5-7335d1715d5c","Type":"ContainerDied","Data":"537aebc6900c1658c5eaecfe10d9cb78b336c1fd4f4efbd3c1990aace5340a2f"}
Oct 30 20:17:47.781672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.781647    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:47.909423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.909328    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca\") pod \"87163a46-7b84-4f72-bbf5-7335d1715d5c\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") "
Oct 30 20:17:47.909602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.909458    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca\") pod \"87163a46-7b84-4f72-bbf5-7335d1715d5c\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") "
Oct 30 20:17:47.909602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.909514    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw\" (UniqueName: \"kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw\") pod \"87163a46-7b84-4f72-bbf5-7335d1715d5c\" (UID: \"87163a46-7b84-4f72-bbf5-7335d1715d5c\") "
Oct 30 20:17:47.910971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.910936    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca" pod "87163a46-7b84-4f72-bbf5-7335d1715d5c" (UID: "87163a46-7b84-4f72-bbf5-7335d1715d5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:47.913288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.913255    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca" pod "87163a46-7b84-4f72-bbf5-7335d1715d5c" (UID: "87163a46-7b84-4f72-bbf5-7335d1715d5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:47.913548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:47.913511    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw" pod "87163a46-7b84-4f72-bbf5-7335d1715d5c" (UID: "87163a46-7b84-4f72-bbf5-7335d1715d5c"). InnerVolumeSpecName "kube-api-access-4zpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:48.011372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:48.011330    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:48.011372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:48.011363    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87163a46-7b84-4f72-bbf5-7335d1715d5c-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:48.011372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:48.011377    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpxw\" (UniqueName: \"kubernetes.io/projected/87163a46-7b84-4f72-bbf5-7335d1715d5c-kube-api-access-4zpxw\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:48.611029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:48.610996    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"
Oct 30 20:17:48.611029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:48.611019    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq" event={"ID":"87163a46-7b84-4f72-bbf5-7335d1715d5c","Type":"ContainerDied","Data":"9254ed20023c6014e9aaa2a3bcd420e754ed86e8fd24d901876c6aeead8defe0"}
Oct 30 20:17:48.611247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:48.611076    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9254ed20023c6014e9aaa2a3bcd420e754ed86e8fd24d901876c6aeead8defe0"
Oct 30 20:17:51.935401 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:51.935365    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"]
Oct 30 20:17:51.935879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:51.935619    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" containerID="cri-o://0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28" gracePeriod=30
Oct 30 20:17:51.986218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:51.986181    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"]
Oct 30 20:17:51.986387 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:51.986369    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" containerID="cri-o://c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b" gracePeriod=30
Oct 30 20:17:52.953377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:52.953343    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"]
Oct 30 20:17:52.953846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:52.953574    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" containerID="cri-o://0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22" gracePeriod=30
Oct 30 20:17:53.257118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:53.257020    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"]
Oct 30 20:17:53.495639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:53.495594    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"]
Oct 30 20:17:53.495855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:53.495818    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" containerID="cri-o://4dc3873877aa16f36463dcc68e3d136c018251cc4adc2c901fd1964cb7e7afb7" gracePeriod=30
Oct 30 20:17:53.650957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:53.650834    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" containerID="cri-o://fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600" gracePeriod=30
Oct 30 20:17:54.282354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:54.282317    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:54.282836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:54.282380    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:54.979034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:54.979001    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:54.979216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:54.979052    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:56.225971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:56.225929    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:56.226360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:56.225997    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:56.570870 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:56.570756    4917 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:56.571023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:56.570863    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:57.292438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.292410    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:17:57.331998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.331968    4917 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:57.332142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.332026    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:57.337670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.337649    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:17:57.371010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.370983    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371023    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371053    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371079    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371116    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371145    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371244    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371273    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.371359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371286    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371323    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371365    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371394    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371442    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371484    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371512    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371598    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data\") pod \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\" (UID: \"4ae71a5c-9975-46d6-8a0d-5392b006b42f\") "
Oct 30 20:17:57.371955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371690    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371732    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.371955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371848    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert\") pod \"6c6a27ec-5567-4684-b862-478385943f99\" (UID: \"6c6a27ec-5567-4684-b862-478385943f99\") "
Oct 30 20:17:57.372118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371364    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371399    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371562    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371577    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.371934    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:57.372118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.372017    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.372211    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.372260    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.372637 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.372577    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:57.373439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373412    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6a27ec-5567-4684-b862-478385943f99-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373444    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373457    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373470    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373482    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373544    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373560    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373574    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.373596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373598    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.374061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373612    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.374061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.373933    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.374192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.374041    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:57.374192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.374133    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.374495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.374466    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj" pod "4ae71a5c-9975-46d6-8a0d-5392b006b42f" (UID: "4ae71a5c-9975-46d6-8a0d-5392b006b42f"). InnerVolumeSpecName "kube-api-access-xxksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.374495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.374482    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "kube-api-access-n46kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.374595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.374530    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:57.375264 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.375244    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.375423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.375406    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy" pod "6c6a27ec-5567-4684-b862-478385943f99" (UID: "6c6a27ec-5567-4684-b862-478385943f99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.475986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.475910    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.475986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.475944    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.475986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.475957    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6a27ec-5567-4684-b862-478385943f99-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.475986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.475969    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxksj\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-kube-api-access-xxksj\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.475986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.475984    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n46kg\" (UniqueName: \"kubernetes.io/projected/6c6a27ec-5567-4684-b862-478385943f99-kube-api-access-n46kg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.476258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.476000    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6a27ec-5567-4684-b862-478385943f99-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.476258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.476013    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.476258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.476024    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae71a5c-9975-46d6-8a0d-5392b006b42f-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.684109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.684067    4917 generic.go:356] "Generic (PLEG): container finished" podID="6c6a27ec-5567-4684-b862-478385943f99" containerID="0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28" exitCode=0
Oct 30 20:17:57.684270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.684194    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"
Oct 30 20:17:57.684341 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.684278    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" event={"ID":"6c6a27ec-5567-4684-b862-478385943f99","Type":"ContainerDied","Data":"0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28"}
Oct 30 20:17:57.684341 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.684330    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7" event={"ID":"6c6a27ec-5567-4684-b862-478385943f99","Type":"ContainerDied","Data":"33b5b58580ca9bba57709dc1e1b52735e2dbff79a2f9bdac5ec0b7401e6e2a0b"}
Oct 30 20:17:57.684417 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.684362    4917 scope.go:122] "RemoveContainer" containerID="0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28"
Oct 30 20:17:57.686383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.686355    4917 generic.go:356] "Generic (PLEG): container finished" podID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerID="c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b" exitCode=0
Oct 30 20:17:57.686563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.686537    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" event={"ID":"4ae71a5c-9975-46d6-8a0d-5392b006b42f","Type":"ContainerDied","Data":"c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b"}
Oct 30 20:17:57.686699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.686586    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln" event={"ID":"4ae71a5c-9975-46d6-8a0d-5392b006b42f","Type":"ContainerDied","Data":"ab3636f0b193eb7c4cd34b1261e501a9e43a658350728e5a514b764bcf7443ca"}
Oct 30 20:17:57.686699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.686662    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"
Oct 30 20:17:57.699710 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.699559    4917 scope.go:122] "RemoveContainer" containerID="0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28"
Oct 30 20:17:57.700003 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:57.699937    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28\": container with ID starting with 0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28 not found: ID does not exist" containerID="0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28"
Oct 30 20:17:57.700003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.699977    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28"} err="failed to get container status \"0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28\": rpc error: code = NotFound desc = could not find container \"0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28\": container with ID starting with 0d1bbd57a7e5fadc0bd9c933f514d6082a07a345da2462cb468c4ed62b1a9a28 not found: ID does not exist"
Oct 30 20:17:57.700003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.700002    4917 scope.go:122] "RemoveContainer" containerID="c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b"
Oct 30 20:17:57.710236 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.710216    4917 scope.go:122] "RemoveContainer" containerID="c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b"
Oct 30 20:17:57.710564 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:57.710546    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b\": container with ID starting with c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b not found: ID does not exist" containerID="c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b"
Oct 30 20:17:57.710624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.710570    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b"} err="failed to get container status \"c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b\": rpc error: code = NotFound desc = could not find container \"c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b\": container with ID starting with c9ec87eb213812bdcab577e15975074cd577620d90b732450ecad0a0c44d807b not found: ID does not exist"
Oct 30 20:17:57.925979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.925848    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"]
Oct 30 20:17:57.974873 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:57.965890    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-bgwln"]
Oct 30 20:17:58.019568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.019537    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" path="/var/lib/kubelet/pods/4ae71a5c-9975-46d6-8a0d-5392b006b42f/volumes"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.020055    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"]
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021151    4917 cpu_manager.go:414] "containerMap: removing container" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021173    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021234    4917 cpu_manager.go:414] "containerMap: removing container" podUID="87163a46-7b84-4f72-bbf5-7335d1715d5c" containerName="revoke-keys"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021243    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87163a46-7b84-4f72-bbf5-7335d1715d5c" containerName="revoke-keys"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021261    4917 cpu_manager.go:414] "containerMap: removing container" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021269    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021428    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae71a5c-9975-46d6-8a0d-5392b006b42f" containerName="istio-proxy"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021451    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6a27ec-5567-4684-b862-478385943f99" containerName="istio-proxy"
Oct 30 20:17:58.028853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.021472    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="87163a46-7b84-4f72-bbf5-7335d1715d5c" containerName="revoke-keys"
Oct 30 20:17:58.031084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.031057    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"]
Oct 30 20:17:58.031230 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.031157    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"]
Oct 30 20:17:58.031230 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.031205    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-smzf7"]
Oct 30 20:17:58.031230 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.031165    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.084866 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.084838    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:58.085011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.084838    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:17:58.085011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.084853    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:17:58.085011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.084879    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:17:58.088469 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.088443    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.088780 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.088740    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.088994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.088938    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqsv\" (UniqueName: \"kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.190428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.190391    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.190634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.190547    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqsv\" (UniqueName: \"kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.190719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.190698    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.191639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.191606    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.197810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.194357    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.254869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.254806    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqsv\" (UniqueName: \"kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv\") pod \"maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.294110 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.294081    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:17:58.394050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394004    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.394280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394064    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.394280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394170    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.394523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394331    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.394661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394613    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.394902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394871    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.395119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.394759    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.395205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.395172    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.395502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.395480    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.395601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.395555    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.395989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.395970    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.396487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.396335    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.396937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.396905    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.397288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397178    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.397288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.396675    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket\") pod \"6240e660-a35c-4f25-8831-9156367ae6ad\" (UID: \"6240e660-a35c-4f25-8831-9156367ae6ad\") "
Oct 30 20:17:58.397288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397261    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.397605 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397579    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "kube-api-access-2kmlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.397855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397828    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6240e660-a35c-4f25-8831-9156367ae6ad-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397859    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmlc\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-kube-api-access-2kmlc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397869    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397878    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397886    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397895    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397903    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6240e660-a35c-4f25-8831-9156367ae6ad-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.397942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.397911    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6240e660-a35c-4f25-8831-9156367ae6ad-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.398301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.398017    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo" pod "6240e660-a35c-4f25-8831-9156367ae6ad" (UID: "6240e660-a35c-4f25-8831-9156367ae6ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:58.401463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.401443    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:17:58.499985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.499943    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6240e660-a35c-4f25-8831-9156367ae6ad-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.697902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.697864    4917 generic.go:356] "Generic (PLEG): container finished" podID="6240e660-a35c-4f25-8831-9156367ae6ad" containerID="0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22" exitCode=0
Oct 30 20:17:58.698160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.698013    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" event={"ID":"6240e660-a35c-4f25-8831-9156367ae6ad","Type":"ContainerDied","Data":"0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22"}
Oct 30 20:17:58.698160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.698070    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn" event={"ID":"6240e660-a35c-4f25-8831-9156367ae6ad","Type":"ContainerDied","Data":"5bec3d187d06e8a8feb556ab433d1fcf4afa88d4904d4162bd06de82855fe07f"}
Oct 30 20:17:58.698160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.698101    4917 scope.go:122] "RemoveContainer" containerID="0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22"
Oct 30 20:17:58.698409 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.698205    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"
Oct 30 20:17:58.700858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.700749    4917 generic.go:356] "Generic (PLEG): container finished" podID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerID="4dc3873877aa16f36463dcc68e3d136c018251cc4adc2c901fd1964cb7e7afb7" exitCode=0
Oct 30 20:17:58.700935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.700897    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" event={"ID":"d52b2056-acf0-4ab9-be03-f354baf75ba3","Type":"ContainerDied","Data":"4dc3873877aa16f36463dcc68e3d136c018251cc4adc2c901fd1964cb7e7afb7"}
Oct 30 20:17:58.765987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.765956    4917 scope.go:122] "RemoveContainer" containerID="0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22"
Oct 30 20:17:58.766266 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:58.766247    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22\": container with ID starting with 0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22 not found: ID does not exist" containerID="0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22"
Oct 30 20:17:58.766343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.766274    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22"} err="failed to get container status \"0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22\": rpc error: code = NotFound desc = could not find container \"0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22\": container with ID starting with 0ba3fa97f64a45de46356eaa793842f11c337d6c83f348b5fd9d645e733e0e22 not found: ID does not exist"
Oct 30 20:17:58.766413 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.766401    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:17:58.792006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.791879    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"]
Oct 30 20:17:58.799032 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:17:58.798995    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd203bac0_dbbd_41be_a657_604990c80865.slice/crio-746d208910663394996e0cd27ca24ac2a3560e7a48603d33bc53d6d213548ee7 WatchSource:0}: Error finding container 746d208910663394996e0cd27ca24ac2a3560e7a48603d33bc53d6d213548ee7: Status 404 returned error can't find the container with id 746d208910663394996e0cd27ca24ac2a3560e7a48603d33bc53d6d213548ee7
Oct 30 20:17:58.803516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803447    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.803652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803634    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.803730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803700    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.803854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803838    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.803941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803875    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.803941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803935    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.804061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.803997    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.804061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804009    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.804061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804026    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.804061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804046    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.804274 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804055    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data\") pod \"d52b2056-acf0-4ab9-be03-f354baf75ba3\" (UID: \"d52b2056-acf0-4ab9-be03-f354baf75ba3\") "
Oct 30 20:17:58.804331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804267    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.804331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804301    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.804493 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804474    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.804700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804676    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52b2056-acf0-4ab9-be03-f354baf75ba3-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.804819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804713    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.804819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804739    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.804819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.804759    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.805093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.805065    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.808114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.808085    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.808193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.808117    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:58.808193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.808138    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.810899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.810859    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v" pod "d52b2056-acf0-4ab9-be03-f354baf75ba3" (UID: "d52b2056-acf0-4ab9-be03-f354baf75ba3"). InnerVolumeSpecName "kube-api-access-rcx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.871984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.871920    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"]
Oct 30 20:17:58.876838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.876805    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-z7xmn"]
Oct 30 20:17:58.906096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.906064    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcx2v\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-kube-api-access-rcx2v\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.906096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.906096    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.906284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.906106    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.906284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.906115    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52b2056-acf0-4ab9-be03-f354baf75ba3-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.953071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:58.953008    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:17:59.006921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.006891    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.006988    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007056    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007095    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007122    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007204    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007284    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007382    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.007851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.007818    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:59.008394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.008005    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:59.008394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.008016    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:59.008394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.008360    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:59.008993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.008950    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:59.012134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.011641    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.012134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.011664    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.012134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.011678    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.012134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.011691    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.012134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.011704    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621da606-4fd4-48d8-8bba-aacb03f1644f-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.013924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.013853    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:59.014136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.014109    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "kube-api-access-fl65c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:59.015851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.015817    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:59.113161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.113106    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token\") pod \"621da606-4fd4-48d8-8bba-aacb03f1644f\" (UID: \"621da606-4fd4-48d8-8bba-aacb03f1644f\") "
Oct 30 20:17:59.114505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.114455    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.114505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.114494    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.114735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.114509    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl65c\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-kube-api-access-fl65c\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.116855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.116822    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token" pod "621da606-4fd4-48d8-8bba-aacb03f1644f" (UID: "621da606-4fd4-48d8-8bba-aacb03f1644f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:59.216046 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.216017    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621da606-4fd4-48d8-8bba-aacb03f1644f-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:59.716041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.716012    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"
Oct 30 20:17:59.716041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.716025    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj" event={"ID":"d52b2056-acf0-4ab9-be03-f354baf75ba3","Type":"ContainerDied","Data":"2cae848d9966bd9594ca42ba5fe681e95126f3f0c3513d04a5670afb8045b4c2"}
Oct 30 20:17:59.716591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.716088    4917 scope.go:122] "RemoveContainer" containerID="4dc3873877aa16f36463dcc68e3d136c018251cc4adc2c901fd1964cb7e7afb7"
Oct 30 20:17:59.718070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.718047    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz" event={"ID":"d203bac0-dbbd-41be-a657-604990c80865","Type":"ContainerStarted","Data":"93cc485fd465d7959805606cd9780654f7b452d7dda6a610b3102c4a79f55c52"}
Oct 30 20:17:59.718213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.718091    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz" event={"ID":"d203bac0-dbbd-41be-a657-604990c80865","Type":"ContainerStarted","Data":"746d208910663394996e0cd27ca24ac2a3560e7a48603d33bc53d6d213548ee7"}
Oct 30 20:17:59.719570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.719548    4917 generic.go:356] "Generic (PLEG): container finished" podID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerID="fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600" exitCode=0
Oct 30 20:17:59.719707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.719598    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"
Oct 30 20:17:59.719707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.719606    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" event={"ID":"621da606-4fd4-48d8-8bba-aacb03f1644f","Type":"ContainerDied","Data":"fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600"}
Oct 30 20:17:59.719707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.719671    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm" event={"ID":"621da606-4fd4-48d8-8bba-aacb03f1644f","Type":"ContainerDied","Data":"3b5f030c3afcdfcd34f19b9e224a9d299df58eaec05e4f8c782544a4d10defd8"}
Oct 30 20:17:59.729786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.729748    4917 scope.go:122] "RemoveContainer" containerID="fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600"
Oct 30 20:17:59.740712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.740689    4917 scope.go:122] "RemoveContainer" containerID="fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600"
Oct 30 20:17:59.741091 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:17:59.741064    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600\": container with ID starting with fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600 not found: ID does not exist" containerID="fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600"
Oct 30 20:17:59.741177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.741102    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600"} err="failed to get container status \"fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600\": rpc error: code = NotFound desc = could not find container \"fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600\": container with ID starting with fdf948e075fc24eed7bdf55b8f98803a04fa2b4dec12e90a5eee2feccb8d2600 not found: ID does not exist"
Oct 30 20:17:59.888751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:17:59.888700    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz" podStartSLOduration=2.888684806 podStartE2EDuration="2.888684806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:59.887934442 +0000 UTC m=+2176.052336084" watchObservedRunningTime="2026-10-30 20:17:59.888684806 +0000 UTC m=+2176.053086447"
Oct 30 20:18:00.005030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.004958    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" path="/var/lib/kubelet/pods/6240e660-a35c-4f25-8831-9156367ae6ad/volumes"
Oct 30 20:18:00.005504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.005489    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6a27ec-5567-4684-b862-478385943f99" path="/var/lib/kubelet/pods/6c6a27ec-5567-4684-b862-478385943f99/volumes"
Oct 30 20:18:00.005942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.005919    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"]
Oct 30 20:18:00.010455 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.010434    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kfnlm"]
Oct 30 20:18:00.121852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.121816    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"]
Oct 30 20:18:00.128587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.128560    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-pqcnj"]
Oct 30 20:18:00.561907 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.561874    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"]
Oct 30 20:18:00.562454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562440    4917 cpu_manager.go:414] "containerMap: removing container" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy"
Oct 30 20:18:00.562504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562455    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy"
Oct 30 20:18:00.562504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562487    4917 cpu_manager.go:414] "containerMap: removing container" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy"
Oct 30 20:18:00.562504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562493    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy"
Oct 30 20:18:00.562598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562522    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy"
Oct 30 20:18:00.562598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562528    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy"
Oct 30 20:18:00.562660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562637    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" containerName="istio-proxy"
Oct 30 20:18:00.562660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562646    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="6240e660-a35c-4f25-8831-9156367ae6ad" containerName="istio-proxy"
Oct 30 20:18:00.562660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.562658    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" containerName="istio-proxy"
Oct 30 20:18:00.566443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.566423    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.633255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.633223    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.633410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.633300    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9c6c\" (UniqueName: \"kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.633450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.633427    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.648899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.648867    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"]
Oct 30 20:18:00.734363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.734334    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9c6c\" (UniqueName: \"kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.734869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.734551    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.734869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.734744    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.735369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.735339    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.735761 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.735735    4917 generic.go:356] "Generic (PLEG): container finished" podID="d203bac0-dbbd-41be-a657-604990c80865" containerID="93cc485fd465d7959805606cd9780654f7b452d7dda6a610b3102c4a79f55c52" exitCode=0
Oct 30 20:18:00.735881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.735820    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz" event={"ID":"d203bac0-dbbd-41be-a657-604990c80865","Type":"ContainerDied","Data":"93cc485fd465d7959805606cd9780654f7b452d7dda6a610b3102c4a79f55c52"}
Oct 30 20:18:00.737397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.737375    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.796369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.796343    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9c6c\" (UniqueName: \"kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c\") pod \"maas-api-revoke-sub-f115d539-9qgdm\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:00.880380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:00.880295    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:01.262833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.262510    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"]
Oct 30 20:18:01.262833 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:18:01.262565    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81f1fe9_9614_42c6_9d20_0b57ccf851b3.slice/crio-05ac1f7e391d0e7c012e30b97169e2e59c5caf6c7a6ca3bf97c4b643afa0df54 WatchSource:0}: Error finding container 05ac1f7e391d0e7c012e30b97169e2e59c5caf6c7a6ca3bf97c4b643afa0df54: Status 404 returned error can't find the container with id 05ac1f7e391d0e7c012e30b97169e2e59c5caf6c7a6ca3bf97c4b643afa0df54
Oct 30 20:18:01.772329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.772167    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm" event={"ID":"e81f1fe9-9614-42c6-9d20-0b57ccf851b3","Type":"ContainerStarted","Data":"4198e9deedf216f44fcc2e75fae3be4b4f812589213be9af14b1066107872c84"}
Oct 30 20:18:01.772329 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.772245    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm" event={"ID":"e81f1fe9-9614-42c6-9d20-0b57ccf851b3","Type":"ContainerStarted","Data":"05ac1f7e391d0e7c012e30b97169e2e59c5caf6c7a6ca3bf97c4b643afa0df54"}
Oct 30 20:18:01.891049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.890994    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm" podStartSLOduration=1.890978773 podStartE2EDuration="1.890978773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:01.889891463 +0000 UTC m=+2178.054293130" watchObservedRunningTime="2026-10-30 20:18:01.890978773 +0000 UTC m=+2178.055380428"
Oct 30 20:18:01.931197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.931170    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:18:01.960836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.960804    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv\" (UniqueName: \"kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv\") pod \"d203bac0-dbbd-41be-a657-604990c80865\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") "
Oct 30 20:18:01.961023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.960909    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca\") pod \"d203bac0-dbbd-41be-a657-604990c80865\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") "
Oct 30 20:18:01.961023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.960942    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca\") pod \"d203bac0-dbbd-41be-a657-604990c80865\" (UID: \"d203bac0-dbbd-41be-a657-604990c80865\") "
Oct 30 20:18:01.961291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.961264    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca" pod "d203bac0-dbbd-41be-a657-604990c80865" (UID: "d203bac0-dbbd-41be-a657-604990c80865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:01.961572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.961553    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d203bac0-dbbd-41be-a657-604990c80865-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.962992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.962962    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv" pod "d203bac0-dbbd-41be-a657-604990c80865" (UID: "d203bac0-dbbd-41be-a657-604990c80865"). InnerVolumeSpecName "kube-api-access-qdqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:01.963106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:01.963052    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca" pod "d203bac0-dbbd-41be-a657-604990c80865" (UID: "d203bac0-dbbd-41be-a657-604990c80865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:02.008204 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.008172    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621da606-4fd4-48d8-8bba-aacb03f1644f" path="/var/lib/kubelet/pods/621da606-4fd4-48d8-8bba-aacb03f1644f/volumes"
Oct 30 20:18:02.008819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.008802    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52b2056-acf0-4ab9-be03-f354baf75ba3" path="/var/lib/kubelet/pods/d52b2056-acf0-4ab9-be03-f354baf75ba3/volumes"
Oct 30 20:18:02.063289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.063250    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqsv\" (UniqueName: \"kubernetes.io/projected/d203bac0-dbbd-41be-a657-604990c80865-kube-api-access-qdqsv\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:02.063289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.063289    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d203bac0-dbbd-41be-a657-604990c80865-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:02.781293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.781258    4917 generic.go:356] "Generic (PLEG): container finished" podID="e81f1fe9-9614-42c6-9d20-0b57ccf851b3" containerID="4198e9deedf216f44fcc2e75fae3be4b4f812589213be9af14b1066107872c84" exitCode=0
Oct 30 20:18:02.781760 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.781340    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm" event={"ID":"e81f1fe9-9614-42c6-9d20-0b57ccf851b3","Type":"ContainerDied","Data":"4198e9deedf216f44fcc2e75fae3be4b4f812589213be9af14b1066107872c84"}
Oct 30 20:18:02.782903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.782878    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz" event={"ID":"d203bac0-dbbd-41be-a657-604990c80865","Type":"ContainerDied","Data":"746d208910663394996e0cd27ca24ac2a3560e7a48603d33bc53d6d213548ee7"}
Oct 30 20:18:02.782994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.782907    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"
Oct 30 20:18:02.782994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:02.782923    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746d208910663394996e0cd27ca24ac2a3560e7a48603d33bc53d6d213548ee7"
Oct 30 20:18:03.949846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.949812    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:03.989958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.989929    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca\") pod \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") "
Oct 30 20:18:03.990117 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.989987    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c\" (UniqueName: \"kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c\") pod \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") "
Oct 30 20:18:03.990203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.990182    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca\") pod \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\" (UID: \"e81f1fe9-9614-42c6-9d20-0b57ccf851b3\") "
Oct 30 20:18:03.990308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.990282    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca" pod "e81f1fe9-9614-42c6-9d20-0b57ccf851b3" (UID: "e81f1fe9-9614-42c6-9d20-0b57ccf851b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:03.990656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.990636    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:03.992313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.992287    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c" pod "e81f1fe9-9614-42c6-9d20-0b57ccf851b3" (UID: "e81f1fe9-9614-42c6-9d20-0b57ccf851b3"). InnerVolumeSpecName "kube-api-access-s9c6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:03.992412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:03.992365    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca" pod "e81f1fe9-9614-42c6-9d20-0b57ccf851b3" (UID: "e81f1fe9-9614-42c6-9d20-0b57ccf851b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:04.092440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:04.092345    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:04.092440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:04.092378    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9c6c\" (UniqueName: \"kubernetes.io/projected/e81f1fe9-9614-42c6-9d20-0b57ccf851b3-kube-api-access-s9c6c\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:04.799066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:04.799029    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"
Oct 30 20:18:04.799066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:04.799060    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm" event={"ID":"e81f1fe9-9614-42c6-9d20-0b57ccf851b3","Type":"ContainerDied","Data":"05ac1f7e391d0e7c012e30b97169e2e59c5caf6c7a6ca3bf97c4b643afa0df54"}
Oct 30 20:18:04.799272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:04.799107    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05ac1f7e391d0e7c012e30b97169e2e59c5caf6c7a6ca3bf97c4b643afa0df54"
Oct 30 20:18:08.513342 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:08.513307    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"]
Oct 30 20:18:08.563615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:08.563580    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7961ed-g8mzr"]
Oct 30 20:18:09.680751 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:18:09.680704    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:18:10.004693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:10.004618    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36ce220-5773-4884-b5c8-7913c443fdaa" path="/var/lib/kubelet/pods/b36ce220-5773-4884-b5c8-7913c443fdaa/volumes"
Oct 30 20:18:16.778703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:16.778665    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"]
Oct 30 20:18:16.779181 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:16.778893    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy" containerID="cri-o://f78892d065ef1e95a8b66d41b817ec89be99a0f6d4c8a42b8c18fff516c99212" gracePeriod=30
Oct 30 20:18:17.567328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.567291    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"]
Oct 30 20:18:17.568075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.568053    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e81f1fe9-9614-42c6-9d20-0b57ccf851b3" containerName="revoke-keys"
Oct 30 20:18:17.568075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.568075    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81f1fe9-9614-42c6-9d20-0b57ccf851b3" containerName="revoke-keys"
Oct 30 20:18:17.568270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.568093    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d203bac0-dbbd-41be-a657-604990c80865" containerName="revoke-keys"
Oct 30 20:18:17.568270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.568101    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d203bac0-dbbd-41be-a657-604990c80865" containerName="revoke-keys"
Oct 30 20:18:17.568270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.568244    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81f1fe9-9614-42c6-9d20-0b57ccf851b3" containerName="revoke-keys"
Oct 30 20:18:17.568270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.568269    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d203bac0-dbbd-41be-a657-604990c80865" containerName="revoke-keys"
Oct 30 20:18:17.571729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.571709    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.620338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.620302    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5t89\" (UniqueName: \"kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.620611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.620583    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.620669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.620634    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.640431 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.640403    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:17.661348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.661315    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:17.661497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.661417    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"]
Oct 30 20:18:17.691030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.690997    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:17.691197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.691091    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:18:17.722332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.722300    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5t89\" (UniqueName: \"kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.722535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.722520    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.722611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.722557    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.723183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.723153    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.725067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.725044    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.784116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.784090    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5t89\" (UniqueName: \"kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:17.885188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:17.885109    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:18.266071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:18.266042    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"]
Oct 30 20:18:18.266470 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:18:18.266444    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9670dbc7_15f7_4284_9cac_80a888d123c7.slice/crio-f2a39a034bc04688ac8945e2e95a46e291a70709886e39a866e894c83ad69203 WatchSource:0}: Error finding container f2a39a034bc04688ac8945e2e95a46e291a70709886e39a866e894c83ad69203: Status 404 returned error can't find the container with id f2a39a034bc04688ac8945e2e95a46e291a70709886e39a866e894c83ad69203
Oct 30 20:18:18.911174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:18.911081    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r" event={"ID":"9670dbc7-15f7-4284-9cac-80a888d123c7","Type":"ContainerStarted","Data":"6c893608a542f51b8ceba48f6b6907f30c72580804f12b010ae6c1631e03e6e9"}
Oct 30 20:18:18.911174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:18.911152    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r" event={"ID":"9670dbc7-15f7-4284-9cac-80a888d123c7","Type":"ContainerStarted","Data":"f2a39a034bc04688ac8945e2e95a46e291a70709886e39a866e894c83ad69203"}
Oct 30 20:18:19.031624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:19.031568    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r" podStartSLOduration=2.031554251 podStartE2EDuration="2.031554251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:19.029931002 +0000 UTC m=+2195.194332642" watchObservedRunningTime="2026-10-30 20:18:19.031554251 +0000 UTC m=+2195.195955891"
Oct 30 20:18:19.920028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:19.919996    4917 generic.go:356] "Generic (PLEG): container finished" podID="9670dbc7-15f7-4284-9cac-80a888d123c7" containerID="6c893608a542f51b8ceba48f6b6907f30c72580804f12b010ae6c1631e03e6e9" exitCode=0
Oct 30 20:18:19.920446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:19.920079    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r" event={"ID":"9670dbc7-15f7-4284-9cac-80a888d123c7","Type":"ContainerDied","Data":"6c893608a542f51b8ceba48f6b6907f30c72580804f12b010ae6c1631e03e6e9"}
Oct 30 20:18:21.087328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.087302    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:21.169507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.169475    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89\" (UniqueName: \"kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89\") pod \"9670dbc7-15f7-4284-9cac-80a888d123c7\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") "
Oct 30 20:18:21.169682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.169675    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca\") pod \"9670dbc7-15f7-4284-9cac-80a888d123c7\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") "
Oct 30 20:18:21.169725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.169700    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca\") pod \"9670dbc7-15f7-4284-9cac-80a888d123c7\" (UID: \"9670dbc7-15f7-4284-9cac-80a888d123c7\") "
Oct 30 20:18:21.170057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.170025    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca" pod "9670dbc7-15f7-4284-9cac-80a888d123c7" (UID: "9670dbc7-15f7-4284-9cac-80a888d123c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:21.170289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.170272    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.171612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.171588    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89" pod "9670dbc7-15f7-4284-9cac-80a888d123c7" (UID: "9670dbc7-15f7-4284-9cac-80a888d123c7"). InnerVolumeSpecName "kube-api-access-h5t89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:21.171833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.171807    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca" pod "9670dbc7-15f7-4284-9cac-80a888d123c7" (UID: "9670dbc7-15f7-4284-9cac-80a888d123c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:21.272067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.272033    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9670dbc7-15f7-4284-9cac-80a888d123c7-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.272067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.272063    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5t89\" (UniqueName: \"kubernetes.io/projected/9670dbc7-15f7-4284-9cac-80a888d123c7-kube-api-access-h5t89\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.939003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.938966    4917 generic.go:356] "Generic (PLEG): container finished" podID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerID="f78892d065ef1e95a8b66d41b817ec89be99a0f6d4c8a42b8c18fff516c99212" exitCode=0
Oct 30 20:18:21.939172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.939037    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" event={"ID":"e280fd88-2d98-449d-ab3d-55ad7bf216f8","Type":"ContainerDied","Data":"f78892d065ef1e95a8b66d41b817ec89be99a0f6d4c8a42b8c18fff516c99212"}
Oct 30 20:18:21.939172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.939101    4917 scope.go:122] "RemoveContainer" containerID="1365b76ba70512b221b6bed52067904bb3e199c3f0cd90cf026016a4bf25a726"
Oct 30 20:18:21.940645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.940628    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"
Oct 30 20:18:21.940725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.940676    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r" event={"ID":"9670dbc7-15f7-4284-9cac-80a888d123c7","Type":"ContainerDied","Data":"f2a39a034bc04688ac8945e2e95a46e291a70709886e39a866e894c83ad69203"}
Oct 30 20:18:21.940797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:21.940725    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a39a034bc04688ac8945e2e95a46e291a70709886e39a866e894c83ad69203"
Oct 30 20:18:22.036742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.036718    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:18:22.083408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083378    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083452    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083477    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083507    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083534    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083555    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083592    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083647    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083702    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo\") pod \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\" (UID: \"e280fd88-2d98-449d-ab3d-55ad7bf216f8\") "
Oct 30 20:18:22.083982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083756    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:22.083982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083837    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:22.083982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083933    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:22.083982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.083946    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:22.084187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.084062    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:22.084727 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.084700    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.084888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.084729    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.084888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.084747    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.084888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.084762    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.084888 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.084814    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.085881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.085856    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:22.086092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.086068    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "kube-api-access-ppt57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:22.086177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.086161    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:22.086220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.086166    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token" pod "e280fd88-2d98-449d-ab3d-55ad7bf216f8" (UID: "e280fd88-2d98-449d-ab3d-55ad7bf216f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:22.185861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.185823    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.185861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.185861    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppt57\" (UniqueName: \"kubernetes.io/projected/e280fd88-2d98-449d-ab3d-55ad7bf216f8-kube-api-access-ppt57\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.186263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.185877    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.186263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.185890    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280fd88-2d98-449d-ab3d-55ad7bf216f8-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:22.948990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.948955    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm" event={"ID":"e280fd88-2d98-449d-ab3d-55ad7bf216f8","Type":"ContainerDied","Data":"b6fe70c1c4143cd3fe50a06850fc1ccd0853a2dbc6034222f565cbc92efede43"}
Oct 30 20:18:22.949198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.949014    4917 scope.go:122] "RemoveContainer" containerID="f78892d065ef1e95a8b66d41b817ec89be99a0f6d4c8a42b8c18fff516c99212"
Oct 30 20:18:22.949198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:22.948969    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"
Oct 30 20:18:23.123056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:23.122907    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"]
Oct 30 20:18:23.202144 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:23.202045    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7961ed-openshift-default-59b7d688d9-d2xsm"]
Oct 30 20:18:23.473123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:23.473045    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:18:24.004634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:24.004601    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" path="/var/lib/kubelet/pods/e280fd88-2d98-449d-ab3d-55ad7bf216f8/volumes"
Oct 30 20:18:31.235612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:31.235576    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"]
Oct 30 20:18:31.285752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:31.285723    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3736ef9-bglkx"]
Oct 30 20:18:32.003647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:32.003617    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b401f60-4e13-47fe-afd7-1ec533a659fd" path="/var/lib/kubelet/pods/4b401f60-4e13-47fe-afd7-1ec533a659fd/volumes"
Oct 30 20:18:38.751560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:38.751526    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"]
Oct 30 20:18:38.801988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:38.801959    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fcfa-rmrcq"]
Oct 30 20:18:39.687410 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:18:39.687368    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:18:40.005271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:40.005188    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9dbc411-d8a6-4aa4-9bbd-d5ebed916379" path="/var/lib/kubelet/pods/f9dbc411-d8a6-4aa4-9bbd-d5ebed916379/volumes"
Oct 30 20:18:45.011958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.011922    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"]
Oct 30 20:18:45.012532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012517    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy"
Oct 30 20:18:45.012581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012533    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy"
Oct 30 20:18:45.012581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012553    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy"
Oct 30 20:18:45.012581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012558    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy"
Oct 30 20:18:45.012684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012586    4917 cpu_manager.go:414] "containerMap: removing container" podUID="9670dbc7-15f7-4284-9cac-80a888d123c7" containerName="revoke-keys"
Oct 30 20:18:45.012684 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012591    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9670dbc7-15f7-4284-9cac-80a888d123c7" containerName="revoke-keys"
Oct 30 20:18:45.012752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012687    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="9670dbc7-15f7-4284-9cac-80a888d123c7" containerName="revoke-keys"
Oct 30 20:18:45.012752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.012700    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy"
Oct 30 20:18:45.016354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.016334    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.078147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.078116    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"]
Oct 30 20:18:45.126118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.126082    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:45.180203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.180164    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:45.180548 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.180530    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:45.180801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.180785    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:18:45.197933 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.186956    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.200040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.200013    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xd6v\" (UniqueName: \"kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.200282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.200262    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.247442 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.247405    4917 scope.go:122] "RemoveContainer" containerID="4f68ca8563a8cc456bebb246fedc0b358e0c041e2820727c81fff9b145ae5fe0"
Oct 30 20:18:45.304669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.303788    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.304669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.304097    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.304669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.304155    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xd6v\" (UniqueName: \"kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.304913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.304762    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.310412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.310367    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.366077 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.366050    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xd6v\" (UniqueName: \"kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v\") pod \"maas-api-revoke-sub-c713b5ac-c6txr\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:45.630272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:45.630194    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:46.019321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:46.019289    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"]
Oct 30 20:18:46.020984 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:18:46.020950    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea5515d_53ac_47a8_883b_7d22385252f7.slice/crio-43a531a502fd402462eb8cc593d561dd55bd4586ac07a90eee9037d8a5b62da9 WatchSource:0}: Error finding container 43a531a502fd402462eb8cc593d561dd55bd4586ac07a90eee9037d8a5b62da9: Status 404 returned error can't find the container with id 43a531a502fd402462eb8cc593d561dd55bd4586ac07a90eee9037d8a5b62da9
Oct 30 20:18:46.283288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:46.283203    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr" event={"ID":"fea5515d-53ac-47a8-883b-7d22385252f7","Type":"ContainerStarted","Data":"43a531a502fd402462eb8cc593d561dd55bd4586ac07a90eee9037d8a5b62da9"}
Oct 30 20:18:46.479288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:46.479256    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"]
Oct 30 20:18:46.479481 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:46.479457    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy" containerID="cri-o://eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6" gracePeriod=30
Oct 30 20:18:47.294069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:47.293977    4917 generic.go:356] "Generic (PLEG): container finished" podID="fea5515d-53ac-47a8-883b-7d22385252f7" containerID="9d66280ea606d0028fb98e5e7563f02683d08f12abe3c0f0a07ad636b180aa7f" exitCode=0
Oct 30 20:18:47.294069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:47.294058    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr" event={"ID":"fea5515d-53ac-47a8-883b-7d22385252f7","Type":"ContainerDied","Data":"9d66280ea606d0028fb98e5e7563f02683d08f12abe3c0f0a07ad636b180aa7f"}
Oct 30 20:18:48.464357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.464329    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:48.595063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.594884    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca\") pod \"fea5515d-53ac-47a8-883b-7d22385252f7\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") "
Oct 30 20:18:48.595063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.594973    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca\") pod \"fea5515d-53ac-47a8-883b-7d22385252f7\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") "
Oct 30 20:18:48.595444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.595194    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v\" (UniqueName: \"kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v\") pod \"fea5515d-53ac-47a8-883b-7d22385252f7\" (UID: \"fea5515d-53ac-47a8-883b-7d22385252f7\") "
Oct 30 20:18:48.595489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.595460    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca" pod "fea5515d-53ac-47a8-883b-7d22385252f7" (UID: "fea5515d-53ac-47a8-883b-7d22385252f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:48.595734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.595713    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.597707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.597678    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca" pod "fea5515d-53ac-47a8-883b-7d22385252f7" (UID: "fea5515d-53ac-47a8-883b-7d22385252f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:48.597938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.597916    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v" pod "fea5515d-53ac-47a8-883b-7d22385252f7" (UID: "fea5515d-53ac-47a8-883b-7d22385252f7"). InnerVolumeSpecName "kube-api-access-2xd6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:48.697267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.697220    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xd6v\" (UniqueName: \"kubernetes.io/projected/fea5515d-53ac-47a8-883b-7d22385252f7-kube-api-access-2xd6v\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:48.697267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:48.697258    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea5515d-53ac-47a8-883b-7d22385252f7-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:49.441127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:49.441086    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr" event={"ID":"fea5515d-53ac-47a8-883b-7d22385252f7","Type":"ContainerDied","Data":"43a531a502fd402462eb8cc593d561dd55bd4586ac07a90eee9037d8a5b62da9"}
Oct 30 20:18:49.441663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:49.441635    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a531a502fd402462eb8cc593d561dd55bd4586ac07a90eee9037d8a5b62da9"
Oct 30 20:18:49.441841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:49.441432    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"
Oct 30 20:18:51.756511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.756446    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:18:51.785536 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785507    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.785732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785711    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.785817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785766    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.785862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785846    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.785918 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785901    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.785963 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785944    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.786009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.785954    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:51.786102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786064    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786151    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.786229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786158    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786167    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786187    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.786379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786234    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo\") pod \"795942c1-11a0-4218-a931-a6a0807d8adf\" (UID: \"795942c1-11a0-4218-a931-a6a0807d8adf\") "
Oct 30 20:18:51.786379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786338    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.786731 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786711    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.786731 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786732    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.786924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786742    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.786924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786751    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.786924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.786759    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795942c1-11a0-4218-a931-a6a0807d8adf-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.788227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.788199    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:51.788354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.788252    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:51.788521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.788505    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "kube-api-access-rgppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:51.788575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.788512    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo" pod "795942c1-11a0-4218-a931-a6a0807d8adf" (UID: "795942c1-11a0-4218-a931-a6a0807d8adf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:51.887615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.887580    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795942c1-11a0-4218-a931-a6a0807d8adf-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.887615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.887611    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.887867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.887624    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795942c1-11a0-4218-a931-a6a0807d8adf-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:51.887867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:51.887637    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgppq\" (UniqueName: \"kubernetes.io/projected/795942c1-11a0-4218-a931-a6a0807d8adf-kube-api-access-rgppq\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:52.471137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.471102    4917 generic.go:356] "Generic (PLEG): container finished" podID="795942c1-11a0-4218-a931-a6a0807d8adf" containerID="eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6" exitCode=0
Oct 30 20:18:52.471303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.471154    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"
Oct 30 20:18:52.471303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.471166    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" event={"ID":"795942c1-11a0-4218-a931-a6a0807d8adf","Type":"ContainerDied","Data":"eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6"}
Oct 30 20:18:52.471303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.471211    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k" event={"ID":"795942c1-11a0-4218-a931-a6a0807d8adf","Type":"ContainerDied","Data":"0b0e1160ec6c02a70c37760380c05d17d7697868d02563f441e7f87526656711"}
Oct 30 20:18:52.471303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.471233    4917 scope.go:122] "RemoveContainer" containerID="eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6"
Oct 30 20:18:52.482005 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.481987    4917 scope.go:122] "RemoveContainer" containerID="6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"
Oct 30 20:18:52.491087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.491062    4917 scope.go:122] "RemoveContainer" containerID="eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6"
Oct 30 20:18:52.491380 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:18:52.491362    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6\": container with ID starting with eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6 not found: ID does not exist" containerID="eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6"
Oct 30 20:18:52.491441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.491387    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6"} err="failed to get container status \"eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6\": rpc error: code = NotFound desc = could not find container \"eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6\": container with ID starting with eb6a72b514486378ae7a622b3f6f56254f557b28c08b7fe4435114da5b884cf6 not found: ID does not exist"
Oct 30 20:18:52.491441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.491407    4917 scope.go:122] "RemoveContainer" containerID="6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"
Oct 30 20:18:52.491638 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:18:52.491619    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4\": container with ID starting with 6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4 not found: ID does not exist" containerID="6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"
Oct 30 20:18:52.491695 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.491647    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4"} err="failed to get container status \"6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4\": rpc error: code = NotFound desc = could not find container \"6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4\": container with ID starting with 6da2cb9d6fa421fdfe2464d923c8cd27ccc64487f310feebcc5fdd12581383f4 not found: ID does not exist"
Oct 30 20:18:52.641258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.641228    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"]
Oct 30 20:18:52.691523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:52.691492    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-77fcfa-openshift-default-5bf5468857-rnl2k"]
Oct 30 20:18:54.004436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:54.004402    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" path="/var/lib/kubelet/pods/795942c1-11a0-4218-a931-a6a0807d8adf/volumes"
Oct 30 20:18:58.007462 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.007431    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"]
Oct 30 20:18:58.008184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008161    4917 cpu_manager.go:414] "containerMap: removing container" podUID="fea5515d-53ac-47a8-883b-7d22385252f7" containerName="revoke-keys"
Oct 30 20:18:58.008309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008185    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea5515d-53ac-47a8-883b-7d22385252f7" containerName="revoke-keys"
Oct 30 20:18:58.008309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008216    4917 cpu_manager.go:414] "containerMap: removing container" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy"
Oct 30 20:18:58.008309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008223    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy"
Oct 30 20:18:58.008469 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008455    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy"
Oct 30 20:18:58.008524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008470    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e280fd88-2d98-449d-ab3d-55ad7bf216f8" containerName="istio-proxy"
Oct 30 20:18:58.008524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008488    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy"
Oct 30 20:18:58.008524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.008503    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea5515d-53ac-47a8-883b-7d22385252f7" containerName="revoke-keys"
Oct 30 20:18:58.013520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.013485    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.069855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.069819    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.070065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.070047    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4m4\" (UniqueName: \"kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.070215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.070195    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.070363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.070345    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"]
Oct 30 20:18:58.115825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.115765    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:58.115825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.115813    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:58.117149 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.117128    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:58.117256 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.117248    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:18:58.171881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.171851    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4m4\" (UniqueName: \"kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.172055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.171982    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.172124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.172051    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.172647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.172626    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.174447 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.174426    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.234195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.234168    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4m4\" (UniqueName: \"kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.327454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.327380    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:18:58.708148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:58.708113    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"]
Oct 30 20:18:58.709552 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:18:58.709526    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991491ac_cd4c_4dfd_8e61_0d0789145ac3.slice/crio-a31b228594ec595f88326c7ea1ee9bdac4cd85626c143277da2c9de8680e50c1 WatchSource:0}: Error finding container a31b228594ec595f88326c7ea1ee9bdac4cd85626c143277da2c9de8680e50c1: Status 404 returned error can't find the container with id a31b228594ec595f88326c7ea1ee9bdac4cd85626c143277da2c9de8680e50c1
Oct 30 20:18:59.533420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:59.533378    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5" event={"ID":"991491ac-cd4c-4dfd-8e61-0d0789145ac3","Type":"ContainerStarted","Data":"49c199d91de1d424043119f1e6efed2faf6e3b9c7338ee86d31666c73572ca55"}
Oct 30 20:18:59.533837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:59.533440    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5" event={"ID":"991491ac-cd4c-4dfd-8e61-0d0789145ac3","Type":"ContainerStarted","Data":"a31b228594ec595f88326c7ea1ee9bdac4cd85626c143277da2c9de8680e50c1"}
Oct 30 20:18:59.653188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:18:59.653134    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5" podStartSLOduration=2.653120931 podStartE2EDuration="2.653120931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:59.652493255 +0000 UTC m=+2235.816894898" watchObservedRunningTime="2026-10-30 20:18:59.653120931 +0000 UTC m=+2235.817522572"
Oct 30 20:19:00.542217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:00.542179    4917 generic.go:356] "Generic (PLEG): container finished" podID="991491ac-cd4c-4dfd-8e61-0d0789145ac3" containerID="49c199d91de1d424043119f1e6efed2faf6e3b9c7338ee86d31666c73572ca55" exitCode=0
Oct 30 20:19:00.542652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:00.542259    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5" event={"ID":"991491ac-cd4c-4dfd-8e61-0d0789145ac3","Type":"ContainerDied","Data":"49c199d91de1d424043119f1e6efed2faf6e3b9c7338ee86d31666c73572ca55"}
Oct 30 20:19:01.711333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.711292    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:19:01.824324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.824221    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca\") pod \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") "
Oct 30 20:19:01.824324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.824304    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4\" (UniqueName: \"kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4\") pod \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") "
Oct 30 20:19:01.824517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.824382    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca\") pod \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\" (UID: \"991491ac-cd4c-4dfd-8e61-0d0789145ac3\") "
Oct 30 20:19:01.824670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.824648    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca" pod "991491ac-cd4c-4dfd-8e61-0d0789145ac3" (UID: "991491ac-cd4c-4dfd-8e61-0d0789145ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:01.825034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.824999    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:01.826564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.826534    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4" pod "991491ac-cd4c-4dfd-8e61-0d0789145ac3" (UID: "991491ac-cd4c-4dfd-8e61-0d0789145ac3"). InnerVolumeSpecName "kube-api-access-qp4m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:01.826564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.826546    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca" pod "991491ac-cd4c-4dfd-8e61-0d0789145ac3" (UID: "991491ac-cd4c-4dfd-8e61-0d0789145ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:01.926660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.926622    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp4m4\" (UniqueName: \"kubernetes.io/projected/991491ac-cd4c-4dfd-8e61-0d0789145ac3-kube-api-access-qp4m4\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:01.926660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:01.926657    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991491ac-cd4c-4dfd-8e61-0d0789145ac3-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:02.562295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:02.562166    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5" event={"ID":"991491ac-cd4c-4dfd-8e61-0d0789145ac3","Type":"ContainerDied","Data":"a31b228594ec595f88326c7ea1ee9bdac4cd85626c143277da2c9de8680e50c1"}
Oct 30 20:19:02.562295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:02.562237    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31b228594ec595f88326c7ea1ee9bdac4cd85626c143277da2c9de8680e50c1"
Oct 30 20:19:02.563121 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:02.562308    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"
Oct 30 20:19:09.680662 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:19:09.680625    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:19:13.078673 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:13.078639    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:19:20.581793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:20.577327    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"]
Oct 30 20:19:20.621015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:20.620969    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d2c70-snpfz"]
Oct 30 20:19:21.895633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:21.895597    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"]
Oct 30 20:19:21.945865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:21.945834    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-gglnq"]
Oct 30 20:19:22.014969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:22.014935    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87163a46-7b84-4f72-bbf5-7335d1715d5c" path="/var/lib/kubelet/pods/87163a46-7b84-4f72-bbf5-7335d1715d5c/volumes"
Oct 30 20:19:22.015423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:22.015409    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d203bac0-dbbd-41be-a657-604990c80865" path="/var/lib/kubelet/pods/d203bac0-dbbd-41be-a657-604990c80865/volumes"
Oct 30 20:19:23.990544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:23.990506    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"]
Oct 30 20:19:23.991069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:23.990706    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" containerID="cri-o://7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2" gracePeriod=30
Oct 30 20:19:25.957097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:25.957054    4917 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:19:25.957721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:25.957688    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:19:28.903516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:28.903475    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"]
Oct 30 20:19:28.903990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:28.903734    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy" containerID="cri-o://0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934" gracePeriod=30
Oct 30 20:19:29.280977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.280952    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:19:29.373505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373458    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373526    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373575    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373607    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373669    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373750    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373818    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373823    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:29.373908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373871    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.373908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.373906    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy\") pod \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\" (UID: \"405f3a3f-fd98-433c-b9d9-0bda69e77a7b\") "
Oct 30 20:19:29.374175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374040    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:29.374175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374081    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:29.374175 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374140    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:29.374325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374236    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:29.374574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374551    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.374661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374575    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.374661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374584    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.374661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374593    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.374661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.374605    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.376063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.376030    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:19:29.376180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.376159    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:29.376382 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.376355    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:29.376486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.376385    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd" pod "405f3a3f-fd98-433c-b9d9-0bda69e77a7b" (UID: "405f3a3f-fd98-433c-b9d9-0bda69e77a7b"). InnerVolumeSpecName "kube-api-access-kf5xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:29.477932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.477895    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.478093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.477936    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf5xd\" (UniqueName: \"kubernetes.io/projected/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-kube-api-access-kf5xd\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.478093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.477954    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.478093 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.477967    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f3a3f-fd98-433c-b9d9-0bda69e77a7b-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:29.821003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.820915    4917 generic.go:356] "Generic (PLEG): container finished" podID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerID="7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2" exitCode=0
Oct 30 20:19:29.821003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.820962    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"
Oct 30 20:19:29.821261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.821002    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" event={"ID":"405f3a3f-fd98-433c-b9d9-0bda69e77a7b","Type":"ContainerDied","Data":"7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2"}
Oct 30 20:19:29.821261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.821069    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb" event={"ID":"405f3a3f-fd98-433c-b9d9-0bda69e77a7b","Type":"ContainerDied","Data":"8a36d53670ed9d6c4ee16ecb604e3b0bf3d07da793f2aaf842550bafc6980e7c"}
Oct 30 20:19:29.821261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.821099    4917 scope.go:122] "RemoveContainer" containerID="7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2"
Oct 30 20:19:29.832179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.832162    4917 scope.go:122] "RemoveContainer" containerID="7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2"
Oct 30 20:19:29.832484 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:19:29.832461    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2\": container with ID starting with 7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2 not found: ID does not exist" containerID="7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2"
Oct 30 20:19:29.832563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.832489    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2"} err="failed to get container status \"7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2\": rpc error: code = NotFound desc = could not find container \"7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2\": container with ID starting with 7931f66d45e065cc1e496f8a13a30bfe591416acdc19cd4b40c246a01bbee2b2 not found: ID does not exist"
Oct 30 20:19:29.990803 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:29.990747    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"]
Oct 30 20:19:30.040948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:30.040916    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-w4qhb"]
Oct 30 20:19:31.931133 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931097    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"]
Oct 30 20:19:31.931665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931650    4917 cpu_manager.go:414] "containerMap: removing container" podUID="991491ac-cd4c-4dfd-8e61-0d0789145ac3" containerName="revoke-keys"
Oct 30 20:19:31.931708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931666    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991491ac-cd4c-4dfd-8e61-0d0789145ac3" containerName="revoke-keys"
Oct 30 20:19:31.931708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931699    4917 cpu_manager.go:414] "containerMap: removing container" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy"
Oct 30 20:19:31.931708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931704    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy"
Oct 30 20:19:31.931836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931716    4917 cpu_manager.go:414] "containerMap: removing container" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy"
Oct 30 20:19:31.931836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931721    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795942c1-11a0-4218-a931-a6a0807d8adf" containerName="istio-proxy"
Oct 30 20:19:31.931836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931820    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" containerName="istio-proxy"
Oct 30 20:19:31.931836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.931832    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="991491ac-cd4c-4dfd-8e61-0d0789145ac3" containerName="revoke-keys"
Oct 30 20:19:31.937167 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.937149    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:31.990463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:31.990411    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:32.007193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.007165    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405f3a3f-fd98-433c-b9d9-0bda69e77a7b" path="/var/lib/kubelet/pods/405f3a3f-fd98-433c-b9d9-0bda69e77a7b/volumes"
Oct 30 20:19:32.007513 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.007490    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"]
Oct 30 20:19:32.011649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.011627    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.011745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.011666    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.011932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.011908    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8jf\" (UniqueName: \"kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.040496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.040465    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:19:32.040601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.040517    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:32.040865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.040845    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:32.113352 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.113322    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8jf\" (UniqueName: \"kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.113508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.113457    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.113508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.113478    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.114153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.114134    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.115946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.115926    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.148075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.148035    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"]
Oct 30 20:19:32.152388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.152372    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.175965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.175940    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8jf\" (UniqueName: \"kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf\") pod \"maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.212429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.212364    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"]
Oct 30 20:19:32.250467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.250441    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:32.315764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.315732    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.315977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.315952    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2k79\" (UniqueName: \"kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.316047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.315997    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.417466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.417429    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.417660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.417634    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2k79\" (UniqueName: \"kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.417724 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.417677    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.418309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.418281    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.420254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.420225    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.489578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.489489    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2k79\" (UniqueName: \"kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.650506 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.650372    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"]
Oct 30 20:19:32.780369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.780259    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:32.864815 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:32.864258    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj" event={"ID":"324fe3d4-5e0f-4781-866c-f24711d27894","Type":"ContainerStarted","Data":"2f604c0da122253e2e05e7c9ccb9dad38c6757d3025a8ab5ba84130ded9af6ea"}
Oct 30 20:19:33.204085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:33.204021    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"]
Oct 30 20:19:33.204544 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:19:33.204091    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eee8557_6c1a_413c_bbef_9fc4916d83e8.slice/crio-fc102ac8e3fbc4350d7c8f09b33238f6324417e3d0f98dbc9e8a8193c1d1a8ad WatchSource:0}: Error finding container fc102ac8e3fbc4350d7c8f09b33238f6324417e3d0f98dbc9e8a8193c1d1a8ad: Status 404 returned error can't find the container with id fc102ac8e3fbc4350d7c8f09b33238f6324417e3d0f98dbc9e8a8193c1d1a8ad
Oct 30 20:19:33.874044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:33.874005    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v" event={"ID":"6eee8557-6c1a-413c-bbef-9fc4916d83e8","Type":"ContainerStarted","Data":"2c71f141bb6beb16c52a8b83ffcaea4e4040c3cd11fcdaa507a61bb78941348c"}
Oct 30 20:19:33.874213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:33.874070    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v" event={"ID":"6eee8557-6c1a-413c-bbef-9fc4916d83e8","Type":"ContainerStarted","Data":"fc102ac8e3fbc4350d7c8f09b33238f6324417e3d0f98dbc9e8a8193c1d1a8ad"}
Oct 30 20:19:33.875578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:33.875549    4917 generic.go:356] "Generic (PLEG): container finished" podID="324fe3d4-5e0f-4781-866c-f24711d27894" containerID="533399de30d1a1fdc4dcac90f93ab75a24c0a7735ebe1dd89d3fdd49f1f2b3f2" exitCode=0
Oct 30 20:19:33.875696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:33.875614    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj" event={"ID":"324fe3d4-5e0f-4781-866c-f24711d27894","Type":"ContainerDied","Data":"533399de30d1a1fdc4dcac90f93ab75a24c0a7735ebe1dd89d3fdd49f1f2b3f2"}
Oct 30 20:19:33.993104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:33.992998    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v" podStartSLOduration=1.99298462 podStartE2EDuration="1.99298462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:33.991712921 +0000 UTC m=+2270.156114564" watchObservedRunningTime="2026-10-30 20:19:33.99298462 +0000 UTC m=+2270.157386263"
Oct 30 20:19:34.186745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.186720    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:19:34.347586 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347548    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347618    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347668    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347698    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347803    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347919    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.347939    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:34.348263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.348028    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.348399    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.348448    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert\") pod \"93420d97-cb7f-4516-b7bd-75922905beb3\" (UID: \"93420d97-cb7f-4516-b7bd-75922905beb3\") "
Oct 30 20:19:34.348602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.348476    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:34.348705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.348690    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:34.349192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.349152    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:34.349192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.349165    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.349192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.349185    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.349402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.349196    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.351543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.351517    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:34.351543 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.351528    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "kube-api-access-bpx5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:34.351714 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.351567    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:19:34.351714 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.351582    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:34.353974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.353951    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket" pod "93420d97-cb7f-4516-b7bd-75922905beb3" (UID: "93420d97-cb7f-4516-b7bd-75922905beb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:34.450906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.450872    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.450906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.450905    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.451120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.450919    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93420d97-cb7f-4516-b7bd-75922905beb3-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.451120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.450933    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93420d97-cb7f-4516-b7bd-75922905beb3-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.451120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.450945    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpx5f\" (UniqueName: \"kubernetes.io/projected/93420d97-cb7f-4516-b7bd-75922905beb3-kube-api-access-bpx5f\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.451120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.450958    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93420d97-cb7f-4516-b7bd-75922905beb3-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:34.884552 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.884518    4917 generic.go:356] "Generic (PLEG): container finished" podID="6eee8557-6c1a-413c-bbef-9fc4916d83e8" containerID="2c71f141bb6beb16c52a8b83ffcaea4e4040c3cd11fcdaa507a61bb78941348c" exitCode=0
Oct 30 20:19:34.884719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.884604    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v" event={"ID":"6eee8557-6c1a-413c-bbef-9fc4916d83e8","Type":"ContainerDied","Data":"2c71f141bb6beb16c52a8b83ffcaea4e4040c3cd11fcdaa507a61bb78941348c"}
Oct 30 20:19:34.886528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.886502    4917 generic.go:356] "Generic (PLEG): container finished" podID="93420d97-cb7f-4516-b7bd-75922905beb3" containerID="0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934" exitCode=0
Oct 30 20:19:34.886659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.886530    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" event={"ID":"93420d97-cb7f-4516-b7bd-75922905beb3","Type":"ContainerDied","Data":"0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934"}
Oct 30 20:19:34.886659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.886554    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"
Oct 30 20:19:34.886659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.886589    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs" event={"ID":"93420d97-cb7f-4516-b7bd-75922905beb3","Type":"ContainerDied","Data":"dda78a4071e72f6ce2cae9f4387d5395f179067ebec8e9455eb1386b2ecabe49"}
Oct 30 20:19:34.886659 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.886618    4917 scope.go:122] "RemoveContainer" containerID="0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934"
Oct 30 20:19:34.899147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.899127    4917 scope.go:122] "RemoveContainer" containerID="9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"
Oct 30 20:19:34.910104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.910079    4917 scope.go:122] "RemoveContainer" containerID="0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934"
Oct 30 20:19:34.910418 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:19:34.910395    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934\": container with ID starting with 0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934 not found: ID does not exist" containerID="0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934"
Oct 30 20:19:34.910507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.910427    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934"} err="failed to get container status \"0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934\": rpc error: code = NotFound desc = could not find container \"0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934\": container with ID starting with 0e3ca9f37a2017a93cc06743ca7650040c5d980bf1a9f7f4aea30626a2084934 not found: ID does not exist"
Oct 30 20:19:34.910507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.910445    4917 scope.go:122] "RemoveContainer" containerID="9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"
Oct 30 20:19:34.910736 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:19:34.910713    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95\": container with ID starting with 9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95 not found: ID does not exist" containerID="9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"
Oct 30 20:19:34.910820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:34.910746    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95"} err="failed to get container status \"9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95\": rpc error: code = NotFound desc = could not find container \"9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95\": container with ID starting with 9165d1d6c721630bcc424661a8303458d0eba59a6134eb575065cb3fb2a25c95 not found: ID does not exist"
Oct 30 20:19:35.032977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.032953    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:35.119704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.119666    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"]
Oct 30 20:19:35.127634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.127605    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5d2c70-openshift-default-6c6c86599f-n8wqs"]
Oct 30 20:19:35.159865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.159749    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf\" (UniqueName: \"kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf\") pod \"324fe3d4-5e0f-4781-866c-f24711d27894\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") "
Oct 30 20:19:35.159996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.159857    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca\") pod \"324fe3d4-5e0f-4781-866c-f24711d27894\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") "
Oct 30 20:19:35.159996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.159931    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca\") pod \"324fe3d4-5e0f-4781-866c-f24711d27894\" (UID: \"324fe3d4-5e0f-4781-866c-f24711d27894\") "
Oct 30 20:19:35.160183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.160162    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca" pod "324fe3d4-5e0f-4781-866c-f24711d27894" (UID: "324fe3d4-5e0f-4781-866c-f24711d27894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:35.160838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.160815    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:35.162126 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.162107    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf" pod "324fe3d4-5e0f-4781-866c-f24711d27894" (UID: "324fe3d4-5e0f-4781-866c-f24711d27894"). InnerVolumeSpecName "kube-api-access-mc8jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:35.162444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.162422    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca" pod "324fe3d4-5e0f-4781-866c-f24711d27894" (UID: "324fe3d4-5e0f-4781-866c-f24711d27894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:35.262179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.262146    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324fe3d4-5e0f-4781-866c-f24711d27894-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:35.262341 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.262192    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8jf\" (UniqueName: \"kubernetes.io/projected/324fe3d4-5e0f-4781-866c-f24711d27894-kube-api-access-mc8jf\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:35.896322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.896274    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"
Oct 30 20:19:35.896743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.896320    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj" event={"ID":"324fe3d4-5e0f-4781-866c-f24711d27894","Type":"ContainerDied","Data":"2f604c0da122253e2e05e7c9ccb9dad38c6757d3025a8ab5ba84130ded9af6ea"}
Oct 30 20:19:35.896743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:35.896391    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f604c0da122253e2e05e7c9ccb9dad38c6757d3025a8ab5ba84130ded9af6ea"
Oct 30 20:19:36.008572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.008537    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" path="/var/lib/kubelet/pods/93420d97-cb7f-4516-b7bd-75922905beb3/volumes"
Oct 30 20:19:36.065338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.065306    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:36.173079 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.173003    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca\") pod \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") "
Oct 30 20:19:36.173221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.173107    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79\" (UniqueName: \"kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79\") pod \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") "
Oct 30 20:19:36.173221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.173156    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca\") pod \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\" (UID: \"6eee8557-6c1a-413c-bbef-9fc4916d83e8\") "
Oct 30 20:19:36.173417 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.173392    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca" pod "6eee8557-6c1a-413c-bbef-9fc4916d83e8" (UID: "6eee8557-6c1a-413c-bbef-9fc4916d83e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:36.173581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.173561    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:36.175163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.175141    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79" pod "6eee8557-6c1a-413c-bbef-9fc4916d83e8" (UID: "6eee8557-6c1a-413c-bbef-9fc4916d83e8"). InnerVolumeSpecName "kube-api-access-b2k79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:36.175316 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.175298    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca" pod "6eee8557-6c1a-413c-bbef-9fc4916d83e8" (UID: "6eee8557-6c1a-413c-bbef-9fc4916d83e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:36.274520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.274485    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eee8557-6c1a-413c-bbef-9fc4916d83e8-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:36.274520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.274513    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2k79\" (UniqueName: \"kubernetes.io/projected/6eee8557-6c1a-413c-bbef-9fc4916d83e8-kube-api-access-b2k79\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:36.914295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.914257    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"
Oct 30 20:19:36.914820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.914308    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v" event={"ID":"6eee8557-6c1a-413c-bbef-9fc4916d83e8","Type":"ContainerDied","Data":"fc102ac8e3fbc4350d7c8f09b33238f6324417e3d0f98dbc9e8a8193c1d1a8ad"}
Oct 30 20:19:36.914820 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:36.914361    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc102ac8e3fbc4350d7c8f09b33238f6324417e3d0f98dbc9e8a8193c1d1a8ad"
Oct 30 20:19:38.910373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:38.910338    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"]
Oct 30 20:19:38.960395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:38.960358    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb565e-jzd7r"]
Oct 30 20:19:39.680712 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:19:39.680678    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:19:40.003405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:40.003331    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9670dbc7-15f7-4284-9cac-80a888d123c7" path="/var/lib/kubelet/pods/9670dbc7-15f7-4284-9cac-80a888d123c7/volumes"
Oct 30 20:19:48.758762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.758728    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"]
Oct 30 20:19:48.759325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759308    4917 cpu_manager.go:414] "containerMap: removing container" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy"
Oct 30 20:19:48.759365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759325    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy"
Oct 30 20:19:48.759365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759340    4917 cpu_manager.go:414] "containerMap: removing container" podUID="6eee8557-6c1a-413c-bbef-9fc4916d83e8" containerName="revoke-keys"
Oct 30 20:19:48.759365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759345    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eee8557-6c1a-413c-bbef-9fc4916d83e8" containerName="revoke-keys"
Oct 30 20:19:48.759365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759363    4917 cpu_manager.go:414] "containerMap: removing container" podUID="324fe3d4-5e0f-4781-866c-f24711d27894" containerName="revoke-keys"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759368    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324fe3d4-5e0f-4781-866c-f24711d27894" containerName="revoke-keys"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759380    4917 cpu_manager.go:414] "containerMap: removing container" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759384    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759459    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="324fe3d4-5e0f-4781-866c-f24711d27894" containerName="revoke-keys"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759468    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759478    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="93420d97-cb7f-4516-b7bd-75922905beb3" containerName="istio-proxy"
Oct 30 20:19:48.759486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.759484    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eee8557-6c1a-413c-bbef-9fc4916d83e8" containerName="revoke-keys"
Oct 30 20:19:48.764187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.764168    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.822886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.822853    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"]
Oct 30 20:19:48.841660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.841629    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.841875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.841679    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.842012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.841988    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9kx\" (UniqueName: \"kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.867104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.867079    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:48.867221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.867108    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:19:48.867578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.867565    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:48.867677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.867662    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:48.943449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.943419    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9kx\" (UniqueName: \"kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.943637 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.943469    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.943637 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.943524    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.944198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.944175    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:48.945995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:48.945974    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:49.005638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:49.005609    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9kx\" (UniqueName: \"kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx\") pod \"maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:49.078614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:49.078536    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:49.458348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:49.458321    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"]
Oct 30 20:19:49.459754 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:19:49.459724    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e6142d_ad13_4cb3_be7e_5024e4aec2db.slice/crio-f247d207a3da213be5e8245cd3e18b4176c283e7dee56a125f0799b1c19dc56b WatchSource:0}: Error finding container f247d207a3da213be5e8245cd3e18b4176c283e7dee56a125f0799b1c19dc56b: Status 404 returned error can't find the container with id f247d207a3da213be5e8245cd3e18b4176c283e7dee56a125f0799b1c19dc56b
Oct 30 20:19:50.024380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:50.024267    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc" event={"ID":"57e6142d-ad13-4cb3-be7e-5024e4aec2db","Type":"ContainerStarted","Data":"4af13676097c3b4f6fb5d5a48992fe39dfd5eb168ed483f27584c32b6a8f6adc"}
Oct 30 20:19:50.024380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:50.024349    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc" event={"ID":"57e6142d-ad13-4cb3-be7e-5024e4aec2db","Type":"ContainerStarted","Data":"f247d207a3da213be5e8245cd3e18b4176c283e7dee56a125f0799b1c19dc56b"}
Oct 30 20:19:50.141808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:50.141709    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc" podStartSLOduration=2.141692017 podStartE2EDuration="2.141692017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:50.141184419 +0000 UTC m=+2286.305586061" watchObservedRunningTime="2026-10-30 20:19:50.141692017 +0000 UTC m=+2286.306093657"
Oct 30 20:19:51.033013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:51.032982    4917 generic.go:356] "Generic (PLEG): container finished" podID="57e6142d-ad13-4cb3-be7e-5024e4aec2db" containerID="4af13676097c3b4f6fb5d5a48992fe39dfd5eb168ed483f27584c32b6a8f6adc" exitCode=0
Oct 30 20:19:51.033478 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:51.033067    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc" event={"ID":"57e6142d-ad13-4cb3-be7e-5024e4aec2db","Type":"ContainerDied","Data":"4af13676097c3b4f6fb5d5a48992fe39dfd5eb168ed483f27584c32b6a8f6adc"}
Oct 30 20:19:52.221702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.221672    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:52.304878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.304798    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca\") pod \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") "
Oct 30 20:19:52.304878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.304851    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca\") pod \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") "
Oct 30 20:19:52.305078 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.305032    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx\" (UniqueName: \"kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx\") pod \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\" (UID: \"57e6142d-ad13-4cb3-be7e-5024e4aec2db\") "
Oct 30 20:19:52.305217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.305195    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca" pod "57e6142d-ad13-4cb3-be7e-5024e4aec2db" (UID: "57e6142d-ad13-4cb3-be7e-5024e4aec2db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:52.305594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.305575    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:52.306981 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.306956    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca" pod "57e6142d-ad13-4cb3-be7e-5024e4aec2db" (UID: "57e6142d-ad13-4cb3-be7e-5024e4aec2db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:52.307154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.307135    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx" pod "57e6142d-ad13-4cb3-be7e-5024e4aec2db" (UID: "57e6142d-ad13-4cb3-be7e-5024e4aec2db"). InnerVolumeSpecName "kube-api-access-5f9kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:52.406641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.406606    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e6142d-ad13-4cb3-be7e-5024e4aec2db-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:52.406641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.406638    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9kx\" (UniqueName: \"kubernetes.io/projected/57e6142d-ad13-4cb3-be7e-5024e4aec2db-kube-api-access-5f9kx\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:52.817247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.817208    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"]
Oct 30 20:19:52.821170 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:52.821132    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f44ec2-snmq5"]
Oct 30 20:19:53.050556 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:53.050519    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"
Oct 30 20:19:53.050763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:53.050526    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc" event={"ID":"57e6142d-ad13-4cb3-be7e-5024e4aec2db","Type":"ContainerDied","Data":"f247d207a3da213be5e8245cd3e18b4176c283e7dee56a125f0799b1c19dc56b"}
Oct 30 20:19:53.050763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:53.050679    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f247d207a3da213be5e8245cd3e18b4176c283e7dee56a125f0799b1c19dc56b"
Oct 30 20:19:53.386913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:53.386879    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"]
Oct 30 20:19:53.408528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:53.408498    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"]
Oct 30 20:19:54.005461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:19:54.005387    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991491ac-cd4c-4dfd-8e61-0d0789145ac3" path="/var/lib/kubelet/pods/991491ac-cd4c-4dfd-8e61-0d0789145ac3/volumes"
Oct 30 20:20:00.580402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.580365    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"]
Oct 30 20:20:00.651596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.651559    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"]
Oct 30 20:20:00.756966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.756910    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"]
Oct 30 20:20:00.758439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.758408    4917 cpu_manager.go:414] "containerMap: removing container" podUID="57e6142d-ad13-4cb3-be7e-5024e4aec2db" containerName="revoke-keys"
Oct 30 20:20:00.758619 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.758592    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e6142d-ad13-4cb3-be7e-5024e4aec2db" containerName="revoke-keys"
Oct 30 20:20:00.759038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.759014    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e6142d-ad13-4cb3-be7e-5024e4aec2db" containerName="revoke-keys"
Oct 30 20:20:00.764796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.764752    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.820791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.820610    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:00.820791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.820733    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:00.835169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.835076    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.861686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.836062    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:20:00.862420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.836106    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:00.871793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.858224    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"]
Oct 30 20:20:00.872139 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.862308    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.872477 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.872455    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpbj\" (UniqueName: \"kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.981748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.981702    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.987487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.987418    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.987925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.987727    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:00.989370 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:00.989348    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpbj\" (UniqueName: \"kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:01.005833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:01.005569    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:01.051867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:01.051828    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpbj\" (UniqueName: \"kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:01.082136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:01.082105    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:01.472195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:01.472166    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"]
Oct 30 20:20:01.473199 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:20:01.473169    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01b566c_858b_407f_82e8_f96ccd137eae.slice/crio-1d1508cd378859f5ea54aaa5e91fa659439122608824fa076cf70b7c014e3951 WatchSource:0}: Error finding container 1d1508cd378859f5ea54aaa5e91fa659439122608824fa076cf70b7c014e3951: Status 404 returned error can't find the container with id 1d1508cd378859f5ea54aaa5e91fa659439122608824fa076cf70b7c014e3951
Oct 30 20:20:02.139523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:02.139413    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58" event={"ID":"a01b566c-858b-407f-82e8-f96ccd137eae","Type":"ContainerStarted","Data":"ac2b8095a6606563d069e72c878a7ba5c8750a19be81391f5f46b6f6d66399cb"}
Oct 30 20:20:02.139523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:02.139497    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58" event={"ID":"a01b566c-858b-407f-82e8-f96ccd137eae","Type":"ContainerStarted","Data":"1d1508cd378859f5ea54aaa5e91fa659439122608824fa076cf70b7c014e3951"}
Oct 30 20:20:02.260591 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:02.260527    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58" podStartSLOduration=2.260510164 podStartE2EDuration="2.260510164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:02.258203492 +0000 UTC m=+2298.422605132" watchObservedRunningTime="2026-10-30 20:20:02.260510164 +0000 UTC m=+2298.424911810"
Oct 30 20:20:03.150904 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:03.150869    4917 generic.go:356] "Generic (PLEG): container finished" podID="a01b566c-858b-407f-82e8-f96ccd137eae" containerID="ac2b8095a6606563d069e72c878a7ba5c8750a19be81391f5f46b6f6d66399cb" exitCode=0
Oct 30 20:20:03.151314 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:03.150947    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58" event={"ID":"a01b566c-858b-407f-82e8-f96ccd137eae","Type":"ContainerDied","Data":"ac2b8095a6606563d069e72c878a7ba5c8750a19be81391f5f46b6f6d66399cb"}
Oct 30 20:20:04.378987 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.378959    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:04.544643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.544607    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca\") pod \"a01b566c-858b-407f-82e8-f96ccd137eae\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") "
Oct 30 20:20:04.544899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.544834    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj\" (UniqueName: \"kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj\") pod \"a01b566c-858b-407f-82e8-f96ccd137eae\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") "
Oct 30 20:20:04.544899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.544895    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca\") pod \"a01b566c-858b-407f-82e8-f96ccd137eae\" (UID: \"a01b566c-858b-407f-82e8-f96ccd137eae\") "
Oct 30 20:20:04.545253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.545230    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca" pod "a01b566c-858b-407f-82e8-f96ccd137eae" (UID: "a01b566c-858b-407f-82e8-f96ccd137eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:04.545567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.545543    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:04.546886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.546859    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca" pod "a01b566c-858b-407f-82e8-f96ccd137eae" (UID: "a01b566c-858b-407f-82e8-f96ccd137eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:04.547057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.547036    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj" pod "a01b566c-858b-407f-82e8-f96ccd137eae" (UID: "a01b566c-858b-407f-82e8-f96ccd137eae"). InnerVolumeSpecName "kube-api-access-2rpbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:04.647379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.647346    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01b566c-858b-407f-82e8-f96ccd137eae-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:04.647379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:04.647378    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rpbj\" (UniqueName: \"kubernetes.io/projected/a01b566c-858b-407f-82e8-f96ccd137eae-kube-api-access-2rpbj\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.168174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:05.168142    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"
Oct 30 20:20:05.168395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:05.168180    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58" event={"ID":"a01b566c-858b-407f-82e8-f96ccd137eae","Type":"ContainerDied","Data":"1d1508cd378859f5ea54aaa5e91fa659439122608824fa076cf70b7c014e3951"}
Oct 30 20:20:05.168395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:05.168227    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1508cd378859f5ea54aaa5e91fa659439122608824fa076cf70b7c014e3951"
Oct 30 20:20:05.581594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:05.581553    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerName="payload-processing" containerID="cri-o://406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8" gracePeriod=25
Oct 30 20:20:05.652074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:05.652029    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerName="payload-pre-processing" containerID="cri-o://6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b" gracePeriod=25
Oct 30 20:20:08.408150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.408105    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" containerID="cri-o://d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e" gracePeriod=45
Oct 30 20:20:08.430119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.430087    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main" containerID="cri-o://f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5" gracePeriod=45
Oct 30 20:20:08.712035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.712007    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:20:08.739662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.739639    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:20:08.807667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.807620    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home\") pod \"0cc54869-c728-402e-bfd2-3c0caaf05a51\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") "
Oct 30 20:20:08.807869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.807695    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache\") pod \"0cc54869-c728-402e-bfd2-3c0caaf05a51\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") "
Oct 30 20:20:08.807869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.807837    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm\") pod \"0cc54869-c728-402e-bfd2-3c0caaf05a51\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") "
Oct 30 20:20:08.807965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.807950    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs\") pod \"0cc54869-c728-402e-bfd2-3c0caaf05a51\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") "
Oct 30 20:20:08.808013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.807980    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb\") pod \"0cc54869-c728-402e-bfd2-3c0caaf05a51\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") "
Oct 30 20:20:08.808090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.808076    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir\") pod \"0cc54869-c728-402e-bfd2-3c0caaf05a51\" (UID: \"0cc54869-c728-402e-bfd2-3c0caaf05a51\") "
Oct 30 20:20:08.808862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.808835    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir" pod "0cc54869-c728-402e-bfd2-3c0caaf05a51" (UID: "0cc54869-c728-402e-bfd2-3c0caaf05a51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.809028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.809011    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home" pod "0cc54869-c728-402e-bfd2-3c0caaf05a51" (UID: "0cc54869-c728-402e-bfd2-3c0caaf05a51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.809205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.809173    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache" pod "0cc54869-c728-402e-bfd2-3c0caaf05a51" (UID: "0cc54869-c728-402e-bfd2-3c0caaf05a51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.811998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.811972    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm" pod "0cc54869-c728-402e-bfd2-3c0caaf05a51" (UID: "0cc54869-c728-402e-bfd2-3c0caaf05a51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.812243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.812210    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs" pod "0cc54869-c728-402e-bfd2-3c0caaf05a51" (UID: "0cc54869-c728-402e-bfd2-3c0caaf05a51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:08.812437 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.812412    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb" pod "0cc54869-c728-402e-bfd2-3c0caaf05a51" (UID: "0cc54869-c728-402e-bfd2-3c0caaf05a51"). InnerVolumeSpecName "kube-api-access-htwmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:08.909477 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909442    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir\") pod \"51341bce-04de-434a-aea0-a884a76dd5c8\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") "
Oct 30 20:20:08.909681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909526    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home\") pod \"51341bce-04de-434a-aea0-a884a76dd5c8\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") "
Oct 30 20:20:08.909681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909635    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache\") pod \"51341bce-04de-434a-aea0-a884a76dd5c8\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") "
Oct 30 20:20:08.909681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909676    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm\") pod \"51341bce-04de-434a-aea0-a884a76dd5c8\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") "
Oct 30 20:20:08.909901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909700    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs\") pod \"51341bce-04de-434a-aea0-a884a76dd5c8\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") "
Oct 30 20:20:08.909901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909753    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk\" (UniqueName: \"kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk\") pod \"51341bce-04de-434a-aea0-a884a76dd5c8\" (UID: \"51341bce-04de-434a-aea0-a884a76dd5c8\") "
Oct 30 20:20:08.909901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909781    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir" pod "51341bce-04de-434a-aea0-a884a76dd5c8" (UID: "51341bce-04de-434a-aea0-a884a76dd5c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.909901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909806    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home" pod "51341bce-04de-434a-aea0-a884a76dd5c8" (UID: "51341bce-04de-434a-aea0-a884a76dd5c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.910108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.909949    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache" pod "51341bce-04de-434a-aea0-a884a76dd5c8" (UID: "51341bce-04de-434a-aea0-a884a76dd5c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.910353 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910333    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910357    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910370    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910383    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910583 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910421    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910583 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910433    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc54869-c728-402e-bfd2-3c0caaf05a51-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910583 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910443    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwmb\" (UniqueName: \"kubernetes.io/projected/0cc54869-c728-402e-bfd2-3c0caaf05a51-kube-api-access-htwmb\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910583 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910454    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.910583 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.910466    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc54869-c728-402e-bfd2-3c0caaf05a51-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.911951 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.911928    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm" pod "51341bce-04de-434a-aea0-a884a76dd5c8" (UID: "51341bce-04de-434a-aea0-a884a76dd5c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.912049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.912006    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk" pod "51341bce-04de-434a-aea0-a884a76dd5c8" (UID: "51341bce-04de-434a-aea0-a884a76dd5c8"). InnerVolumeSpecName "kube-api-access-ckdnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:08.912250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:08.912221    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs" pod "51341bce-04de-434a-aea0-a884a76dd5c8" (UID: "51341bce-04de-434a-aea0-a884a76dd5c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:09.011730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.011649    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51341bce-04de-434a-aea0-a884a76dd5c8-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.011730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.011687    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51341bce-04de-434a-aea0-a884a76dd5c8-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.011730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.011706    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdnk\" (UniqueName: \"kubernetes.io/projected/51341bce-04de-434a-aea0-a884a76dd5c8-kube-api-access-ckdnk\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.205186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.205152    4917 generic.go:356] "Generic (PLEG): container finished" podID="51341bce-04de-434a-aea0-a884a76dd5c8" containerID="f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5" exitCode=0
Oct 30 20:20:09.205388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.205215    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"
Oct 30 20:20:09.205466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.205403    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" event={"ID":"51341bce-04de-434a-aea0-a884a76dd5c8","Type":"ContainerDied","Data":"f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5"}
Oct 30 20:20:09.205466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.205447    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl" event={"ID":"51341bce-04de-434a-aea0-a884a76dd5c8","Type":"ContainerDied","Data":"3984fd1c0a05322ba7e107f367f9ee88cf17c1bb1bb5d28248da42ddd291fa87"}
Oct 30 20:20:09.205593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.205471    4917 scope.go:122] "RemoveContainer" containerID="f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5"
Oct 30 20:20:09.207875 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.207850    4917 generic.go:356] "Generic (PLEG): container finished" podID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerID="d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e" exitCode=0
Oct 30 20:20:09.208001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.207893    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"
Oct 30 20:20:09.208001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.207977    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" event={"ID":"0cc54869-c728-402e-bfd2-3c0caaf05a51","Type":"ContainerDied","Data":"d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e"}
Oct 30 20:20:09.208189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.208006    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" event={"ID":"0cc54869-c728-402e-bfd2-3c0caaf05a51","Type":"ContainerDied","Data":"009ef69c8c281977fe19e03dd95a98f4338e82eb8f82ed384b66ebaf42d99fcb"}
Oct 30 20:20:09.235292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.235260    4917 scope.go:122] "RemoveContainer" containerID="f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5"
Oct 30 20:20:09.236342 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:09.236287    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5\": container with ID starting with f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5 not found: ID does not exist" containerID="f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5"
Oct 30 20:20:09.236480 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.236348    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5"} err="failed to get container status \"f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5\": rpc error: code = NotFound desc = could not find container \"f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5\": container with ID starting with f5a55844f68f1159c128b1e432fdc85928d8e486ebdac6c2f37f001945fc87f5 not found: ID does not exist"
Oct 30 20:20:09.236480 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.236375    4917 scope.go:122] "RemoveContainer" containerID="d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e"
Oct 30 20:20:09.255939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.253929    4917 scope.go:122] "RemoveContainer" containerID="d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e"
Oct 30 20:20:09.255939 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:09.254615    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e\": container with ID starting with d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e not found: ID does not exist" containerID="d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e"
Oct 30 20:20:09.255939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.254657    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e"} err="failed to get container status \"d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e\": rpc error: code = NotFound desc = could not find container \"d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e\": container with ID starting with d4a584ff889d773230b84e4a42f352f5181bd36399763491e1415180448e5f5e not found: ID does not exist"
Oct 30 20:20:09.376087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.376011    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"]
Oct 30 20:20:09.392640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.392604    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v"]
Oct 30 20:20:09.469950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.469911    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:09.470339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.469916    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:09.504449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.504422    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"]
Oct 30 20:20:09.507792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.507760    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/premium-w3-f84d85-kserve-64447cd98d-ffzcl"]
Oct 30 20:20:09.606290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:09.606250    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-w3-f84d85-kserve-77895ddd7f-lp87v" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:20:09.680242 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:09.680210    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:20:10.003953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:10.003872    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" path="/var/lib/kubelet/pods/0cc54869-c728-402e-bfd2-3c0caaf05a51/volumes"
Oct 30 20:20:10.004498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:10.004472    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" path="/var/lib/kubelet/pods/51341bce-04de-434a-aea0-a884a76dd5c8/volumes"
Oct 30 20:20:11.122863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:11.122823    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"]
Oct 30 20:20:11.172590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:11.172560    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770dfe-dkj6v"]
Oct 30 20:20:12.003319 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:12.003283    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eee8557-6c1a-413c-bbef-9fc4916d83e8" path="/var/lib/kubelet/pods/6eee8557-6c1a-413c-bbef-9fc4916d83e8/volumes"
Oct 30 20:20:19.469727 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.469690    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:19.470136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.469726    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:20:19.642471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.642439    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"]
Oct 30 20:20:19.643010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.642995    4917 cpu_manager.go:414] "containerMap: removing container" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main"
Oct 30 20:20:19.643069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643011    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main"
Oct 30 20:20:19.643069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643027    4917 cpu_manager.go:414] "containerMap: removing container" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main"
Oct 30 20:20:19.643069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643032    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main"
Oct 30 20:20:19.643069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643047    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a01b566c-858b-407f-82e8-f96ccd137eae" containerName="revoke-keys"
Oct 30 20:20:19.643069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643051    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01b566c-858b-407f-82e8-f96ccd137eae" containerName="revoke-keys"
Oct 30 20:20:19.643223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643170    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc54869-c728-402e-bfd2-3c0caaf05a51" containerName="main"
Oct 30 20:20:19.643223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643180    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="51341bce-04de-434a-aea0-a884a76dd5c8" containerName="main"
Oct 30 20:20:19.643223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.643187    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01b566c-858b-407f-82e8-f96ccd137eae" containerName="revoke-keys"
Oct 30 20:20:19.647610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.647591    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.706257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.706226    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"]
Oct 30 20:20:19.750867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.750789    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:19.750867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.750765    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:19.751523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.751508    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:19.751614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.751577    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:20:19.767079 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.767051    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.767216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.767093    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.767358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.767334    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444pc\" (UniqueName: \"kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.869047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.869004    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.869224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.869056    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.869224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.869182    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444pc\" (UniqueName: \"kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.869758 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.869734    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.871547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.871519    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.932371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.932339    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-444pc\" (UniqueName: \"kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:19.961726 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:19.961693    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:20.346742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:20.346708    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"]
Oct 30 20:20:20.347048 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:20:20.347011    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd956f097_ece8_46e1_ac03_3ddb6e2e0cad.slice/crio-26cb5a1a93ff6786b9d4ac282f76ec8bc8f11b3649dc2c6af0295b72f5833707 WatchSource:0}: Error finding container 26cb5a1a93ff6786b9d4ac282f76ec8bc8f11b3649dc2c6af0295b72f5833707: Status 404 returned error can't find the container with id 26cb5a1a93ff6786b9d4ac282f76ec8bc8f11b3649dc2c6af0295b72f5833707
Oct 30 20:20:21.304213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:21.304178    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd" event={"ID":"d956f097-ece8-46e1-ac03-3ddb6e2e0cad","Type":"ContainerStarted","Data":"433af2dfe56a4c1a23e9001b6ebadf0d624637ec9470329150305e8ad1802254"}
Oct 30 20:20:21.304701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:21.304233    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd" event={"ID":"d956f097-ece8-46e1-ac03-3ddb6e2e0cad","Type":"ContainerStarted","Data":"26cb5a1a93ff6786b9d4ac282f76ec8bc8f11b3649dc2c6af0295b72f5833707"}
Oct 30 20:20:21.426026 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:21.425965    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd" podStartSLOduration=2.425950959 podStartE2EDuration="2.425950959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:21.421959377 +0000 UTC m=+2317.586361018" watchObservedRunningTime="2026-10-30 20:20:21.425950959 +0000 UTC m=+2317.590352600"
Oct 30 20:20:22.312396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:22.312362    4917 generic.go:356] "Generic (PLEG): container finished" podID="d956f097-ece8-46e1-ac03-3ddb6e2e0cad" containerID="433af2dfe56a4c1a23e9001b6ebadf0d624637ec9470329150305e8ad1802254" exitCode=0
Oct 30 20:20:22.312857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:22.312456    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd" event={"ID":"d956f097-ece8-46e1-ac03-3ddb6e2e0cad","Type":"ContainerDied","Data":"433af2dfe56a4c1a23e9001b6ebadf0d624637ec9470329150305e8ad1802254"}
Oct 30 20:20:23.487559 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.487533    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:23.617979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.617891    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc\" (UniqueName: \"kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc\") pod \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") "
Oct 30 20:20:23.618162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.618143    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca\") pod \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") "
Oct 30 20:20:23.618835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.618235    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca\") pod \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\" (UID: \"d956f097-ece8-46e1-ac03-3ddb6e2e0cad\") "
Oct 30 20:20:23.618835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.618553    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca" pod "d956f097-ece8-46e1-ac03-3ddb6e2e0cad" (UID: "d956f097-ece8-46e1-ac03-3ddb6e2e0cad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:23.618835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.618803    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:23.620681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.620645    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc" pod "d956f097-ece8-46e1-ac03-3ddb6e2e0cad" (UID: "d956f097-ece8-46e1-ac03-3ddb6e2e0cad"). InnerVolumeSpecName "kube-api-access-444pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:23.633739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.633713    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca" pod "d956f097-ece8-46e1-ac03-3ddb6e2e0cad" (UID: "d956f097-ece8-46e1-ac03-3ddb6e2e0cad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:23.720578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.720544    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:23.720578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:23.720572    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444pc\" (UniqueName: \"kubernetes.io/projected/d956f097-ece8-46e1-ac03-3ddb6e2e0cad-kube-api-access-444pc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:24.329137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:24.329052    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd" event={"ID":"d956f097-ece8-46e1-ac03-3ddb6e2e0cad","Type":"ContainerDied","Data":"26cb5a1a93ff6786b9d4ac282f76ec8bc8f11b3649dc2c6af0295b72f5833707"}
Oct 30 20:20:24.329137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:24.329114    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26cb5a1a93ff6786b9d4ac282f76ec8bc8f11b3649dc2c6af0295b72f5833707"
Oct 30 20:20:24.329137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:24.329065    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"
Oct 30 20:20:24.750383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:24.750344    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"]
Oct 30 20:20:24.750928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:24.750578    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy" containerID="cri-o://a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c" gracePeriod=30
Oct 30 20:20:25.930651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:25.930625    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 20:20:25.967651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:25.967627    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 20:20:26.049108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.049065    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng\" (UniqueName: \"kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng\") pod \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") "
Oct 30 20:20:26.049282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.049123    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck\" (UniqueName: \"kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck\") pod \"ece176ef-2f7f-4348-aa65-82d1150fadf4\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") "
Oct 30 20:20:26.049282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.049233    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume\") pod \"ece176ef-2f7f-4348-aa65-82d1150fadf4\" (UID: \"ece176ef-2f7f-4348-aa65-82d1150fadf4\") "
Oct 30 20:20:26.049425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.049370    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume\") pod \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\" (UID: \"eab70257-8dab-41ea-8a28-2ecf2fc0e790\") "
Oct 30 20:20:26.050095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.050064    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume" pod "ece176ef-2f7f-4348-aa65-82d1150fadf4" (UID: "ece176ef-2f7f-4348-aa65-82d1150fadf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:26.050268 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.050240    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume" pod "eab70257-8dab-41ea-8a28-2ecf2fc0e790" (UID: "eab70257-8dab-41ea-8a28-2ecf2fc0e790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:26.052624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.052598    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng" pod "eab70257-8dab-41ea-8a28-2ecf2fc0e790" (UID: "eab70257-8dab-41ea-8a28-2ecf2fc0e790"). InnerVolumeSpecName "kube-api-access-2gnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:26.053213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.053172    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck" pod "ece176ef-2f7f-4348-aa65-82d1150fadf4" (UID: "ece176ef-2f7f-4348-aa65-82d1150fadf4"). InnerVolumeSpecName "kube-api-access-rmjck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:26.151574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.151497    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gnng\" (UniqueName: \"kubernetes.io/projected/eab70257-8dab-41ea-8a28-2ecf2fc0e790-kube-api-access-2gnng\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:26.151574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.151526    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjck\" (UniqueName: \"kubernetes.io/projected/ece176ef-2f7f-4348-aa65-82d1150fadf4-kube-api-access-rmjck\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:26.151574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.151536    4917 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece176ef-2f7f-4348-aa65-82d1150fadf4-plugins-config-volume\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:26.151574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.151545    4917 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab70257-8dab-41ea-8a28-2ecf2fc0e790-plugins-config-volume\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:26.344120 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.344087    4917 generic.go:356] "Generic (PLEG): container finished" podID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerID="6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b" exitCode=0
Oct 30 20:20:26.344343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.344161    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"
Oct 30 20:20:26.344343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.344174    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" event={"ID":"ece176ef-2f7f-4348-aa65-82d1150fadf4","Type":"ContainerDied","Data":"6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b"}
Oct 30 20:20:26.344343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.344238    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk" event={"ID":"ece176ef-2f7f-4348-aa65-82d1150fadf4","Type":"ContainerDied","Data":"cc6ca0cf0bf5800f56aaa3d51b6977faa6fecd85266c7e99b596cb503fe6c5a9"}
Oct 30 20:20:26.344343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.344277    4917 scope.go:122] "RemoveContainer" containerID="6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b"
Oct 30 20:20:26.345594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.345576    4917 generic.go:356] "Generic (PLEG): container finished" podID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerID="406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8" exitCode=0
Oct 30 20:20:26.345685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.345647    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"
Oct 30 20:20:26.345685 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.345664    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" event={"ID":"eab70257-8dab-41ea-8a28-2ecf2fc0e790","Type":"ContainerDied","Data":"406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8"}
Oct 30 20:20:26.345819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.345713    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9" event={"ID":"eab70257-8dab-41ea-8a28-2ecf2fc0e790","Type":"ContainerDied","Data":"8963d2cb36169e929c9e362ab0dea7e8dd116161eab277b12c1d7d8f66684df2"}
Oct 30 20:20:26.355243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.355219    4917 scope.go:122] "RemoveContainer" containerID="6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b"
Oct 30 20:20:26.355549 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:26.355527    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b\": container with ID starting with 6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b not found: ID does not exist" containerID="6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b"
Oct 30 20:20:26.355626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.355559    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b"} err="failed to get container status \"6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b\": rpc error: code = NotFound desc = could not find container \"6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b\": container with ID starting with 6bd4aadec0ee7c21e6605a332f1ba169bed1bc9210e9a64e7d745c8033badc2b not found: ID does not exist"
Oct 30 20:20:26.355626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.355581    4917 scope.go:122] "RemoveContainer" containerID="406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8"
Oct 30 20:20:26.365539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.365519    4917 scope.go:122] "RemoveContainer" containerID="406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8"
Oct 30 20:20:26.365840 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:26.365816    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8\": container with ID starting with 406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8 not found: ID does not exist" containerID="406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8"
Oct 30 20:20:26.365892 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.365851    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8"} err="failed to get container status \"406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8\": rpc error: code = NotFound desc = could not find container \"406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8\": container with ID starting with 406b3d723e138f1f2ee35fcf9d55d2845e1a62a55947cc5a4b934f15851fd3c8 not found: ID does not exist"
Oct 30 20:20:26.522747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.522716    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"]
Oct 30 20:20:26.537653 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.537617    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f84d85-74678c474b-jdbt9"]
Oct 30 20:20:26.614312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614281    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"]
Oct 30 20:20:26.614845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614830    4917 cpu_manager.go:414] "containerMap: removing container" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerName="payload-processing"
Oct 30 20:20:26.614900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614846    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerName="payload-processing"
Oct 30 20:20:26.614900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614873    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerName="payload-pre-processing"
Oct 30 20:20:26.614900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614879    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerName="payload-pre-processing"
Oct 30 20:20:26.614900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614887    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d956f097-ece8-46e1-ac03-3ddb6e2e0cad" containerName="revoke-keys"
Oct 30 20:20:26.614900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.614892    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d956f097-ece8-46e1-ac03-3ddb6e2e0cad" containerName="revoke-keys"
Oct 30 20:20:26.615059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.615022    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" containerName="payload-processing"
Oct 30 20:20:26.615059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.615034    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" containerName="payload-pre-processing"
Oct 30 20:20:26.615059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.615041    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d956f097-ece8-46e1-ac03-3ddb6e2e0cad" containerName="revoke-keys"
Oct 30 20:20:26.619831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.619810    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.667243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.667213    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"]
Oct 30 20:20:26.677109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.677081    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f84d85-75cffb4f4c-8tfzk"]
Oct 30 20:20:26.686061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.686039    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:26.686224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.686198    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:26.686337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.686304    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:26.686337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.686315    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:20:26.716718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.716683    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"]
Oct 30 20:20:26.758838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.758800    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpl8\" (UniqueName: \"kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.758991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.758962    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.759075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.758989    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.860700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.860596    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.860700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.860647    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.860978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.860859    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpl8\" (UniqueName: \"kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.861293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.861272    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.863146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.863127    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.924458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.924430    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpl8\" (UniqueName: \"kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8\") pod \"maas-api-revoke-sub-2bb79f38-ckl87\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:26.934289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:26.934267    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:27.345640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:27.345613    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"]
Oct 30 20:20:27.346631 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:20:27.346596    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e156de_2914_49d7_8a96_dc624914ea68.slice/crio-d5f8064245966c75c4181ef484ee94d7393d731d558be5f3747c9008bbd08870 WatchSource:0}: Error finding container d5f8064245966c75c4181ef484ee94d7393d731d558be5f3747c9008bbd08870: Status 404 returned error can't find the container with id d5f8064245966c75c4181ef484ee94d7393d731d558be5f3747c9008bbd08870
Oct 30 20:20:28.005108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:28.005073    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab70257-8dab-41ea-8a28-2ecf2fc0e790" path="/var/lib/kubelet/pods/eab70257-8dab-41ea-8a28-2ecf2fc0e790/volumes"
Oct 30 20:20:28.005617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:28.005598    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece176ef-2f7f-4348-aa65-82d1150fadf4" path="/var/lib/kubelet/pods/ece176ef-2f7f-4348-aa65-82d1150fadf4/volumes"
Oct 30 20:20:28.365118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:28.365090    4917 generic.go:356] "Generic (PLEG): container finished" podID="60e156de-2914-49d7-8a96-dc624914ea68" containerID="4f8215345b040b117fe26f2b5f148bdaffaf0d9f63d322f272dd3283de5fffa8" exitCode=0
Oct 30 20:20:28.365287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:28.365172    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87" event={"ID":"60e156de-2914-49d7-8a96-dc624914ea68","Type":"ContainerDied","Data":"4f8215345b040b117fe26f2b5f148bdaffaf0d9f63d322f272dd3283de5fffa8"}
Oct 30 20:20:28.365287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:28.365228    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87" event={"ID":"60e156de-2914-49d7-8a96-dc624914ea68","Type":"ContainerStarted","Data":"d5f8064245966c75c4181ef484ee94d7393d731d558be5f3747c9008bbd08870"}
Oct 30 20:20:29.518815 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.518786    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:29.597397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.597362    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8\" (UniqueName: \"kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8\") pod \"60e156de-2914-49d7-8a96-dc624914ea68\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") "
Oct 30 20:20:29.597601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.597419    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca\") pod \"60e156de-2914-49d7-8a96-dc624914ea68\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") "
Oct 30 20:20:29.597601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.597575    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca\") pod \"60e156de-2914-49d7-8a96-dc624914ea68\" (UID: \"60e156de-2914-49d7-8a96-dc624914ea68\") "
Oct 30 20:20:29.598428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.598398    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca" pod "60e156de-2914-49d7-8a96-dc624914ea68" (UID: "60e156de-2914-49d7-8a96-dc624914ea68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:29.600068 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.600032    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca" pod "60e156de-2914-49d7-8a96-dc624914ea68" (UID: "60e156de-2914-49d7-8a96-dc624914ea68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:29.600192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.600151    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8" pod "60e156de-2914-49d7-8a96-dc624914ea68" (UID: "60e156de-2914-49d7-8a96-dc624914ea68"). InnerVolumeSpecName "kube-api-access-5xpl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:29.699750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.699715    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.699750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.699747    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpl8\" (UniqueName: \"kubernetes.io/projected/60e156de-2914-49d7-8a96-dc624914ea68-kube-api-access-5xpl8\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.699750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:29.699757    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e156de-2914-49d7-8a96-dc624914ea68-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.059177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.059149    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:20:30.225477 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225437    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225488    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225522    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225566    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225605    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225692    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225761    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.225923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225831    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:30.225923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225842    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.226123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.225944    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h\") pod \"a8ebf533-bdbc-4115-b099-69f34261409c\" (UID: \"a8ebf533-bdbc-4115-b099-69f34261409c\") "
Oct 30 20:20:30.226123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.226048    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:30.226608 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.226580    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.226730 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.226612    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.227201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.227174    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:30.227201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.227192    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:30.227419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.227399    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:30.228521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.228496    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:30.228647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.228555    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:20:30.229049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.229025    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "kube-api-access-sr56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:30.229745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.229719    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy" pod "a8ebf533-bdbc-4115-b099-69f34261409c" (UID: "a8ebf533-bdbc-4115-b099-69f34261409c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:30.327656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327558    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.327656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327604    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ebf533-bdbc-4115-b099-69f34261409c-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.327656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327620    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.327656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327634    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.327656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327647    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ebf533-bdbc-4115-b099-69f34261409c-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.328063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327662    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ebf533-bdbc-4115-b099-69f34261409c-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.328063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.327675    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr56h\" (UniqueName: \"kubernetes.io/projected/a8ebf533-bdbc-4115-b099-69f34261409c-kube-api-access-sr56h\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:30.381229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.381195    4917 generic.go:356] "Generic (PLEG): container finished" podID="a8ebf533-bdbc-4115-b099-69f34261409c" containerID="a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c" exitCode=0
Oct 30 20:20:30.381398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.381242    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"
Oct 30 20:20:30.381398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.381284    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" event={"ID":"a8ebf533-bdbc-4115-b099-69f34261409c","Type":"ContainerDied","Data":"a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c"}
Oct 30 20:20:30.381398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.381353    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh" event={"ID":"a8ebf533-bdbc-4115-b099-69f34261409c","Type":"ContainerDied","Data":"de5ec6441b867f5962722cb46414b15965bf69ed57279806b0a7214f0e73c23e"}
Oct 30 20:20:30.381398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.381386    4917 scope.go:122] "RemoveContainer" containerID="a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c"
Oct 30 20:20:30.383006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.382979    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87" event={"ID":"60e156de-2914-49d7-8a96-dc624914ea68","Type":"ContainerDied","Data":"d5f8064245966c75c4181ef484ee94d7393d731d558be5f3747c9008bbd08870"}
Oct 30 20:20:30.383133 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.383013    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"
Oct 30 20:20:30.383133 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.383026    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f8064245966c75c4181ef484ee94d7393d731d558be5f3747c9008bbd08870"
Oct 30 20:20:30.396423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.395477    4917 scope.go:122] "RemoveContainer" containerID="a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c"
Oct 30 20:20:30.396423 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:30.395952    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c\": container with ID starting with a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c not found: ID does not exist" containerID="a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c"
Oct 30 20:20:30.396423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.396007    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c"} err="failed to get container status \"a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c\": rpc error: code = NotFound desc = could not find container \"a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c\": container with ID starting with a803e8a224a64559ed5714b95da18a15a851f3d115d2b840ff1e89538826bd8c not found: ID does not exist"
Oct 30 20:20:30.605148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.605055    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"]
Oct 30 20:20:30.612927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:30.612901    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-wdlbh"]
Oct 30 20:20:32.020861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:32.020827    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" path="/var/lib/kubelet/pods/a8ebf533-bdbc-4115-b099-69f34261409c/volumes"
Oct 30 20:20:38.810838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:38.810794    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"]
Oct 30 20:20:38.811321 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:38.811058    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" containerID="cri-o://2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d" gracePeriod=30
Oct 30 20:20:39.294283 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:39.294247    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"]
Oct 30 20:20:39.301011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:39.300984    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25cc2f-xcfcj"]
Oct 30 20:20:39.680022 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:39.679990    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:20:39.687997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:39.687969    4917 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:20:39.688115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:39.688018    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:20:40.003554 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:40.003479    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324fe3d4-5e0f-4781-866c-f24711d27894" path="/var/lib/kubelet/pods/324fe3d4-5e0f-4781-866c-f24711d27894/volumes"
Oct 30 20:20:40.091571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:40.091537    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:20:44.093258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.093188    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:20:44.214402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214367    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214419    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214491    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214527    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214710 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214564    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214710 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214599    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214710 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214647    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214716    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.214928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.214837    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7\") pod \"68fa4fd4-1346-442b-9781-61233391c11b\" (UID: \"68fa4fd4-1346-442b-9781-61233391c11b\") "
Oct 30 20:20:44.215119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215018    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:44.215119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215078    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:44.215308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215080    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:44.215308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215252    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:44.215565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215535    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.215702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215568    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.215702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215584    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.215702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215600    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.215908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.215764    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:44.216945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.216917    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:44.217031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.216920    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:20:44.217031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.216950    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:44.217214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.217195    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7" pod "68fa4fd4-1346-442b-9781-61233391c11b" (UID: "68fa4fd4-1346-442b-9781-61233391c11b"). InnerVolumeSpecName "kube-api-access-9b7d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:44.316824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.316786    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa4fd4-1346-442b-9781-61233391c11b-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.316824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.316821    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa4fd4-1346-442b-9781-61233391c11b-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.316824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.316831    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.317051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.316840    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa4fd4-1346-442b-9781-61233391c11b-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.317051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.316849    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7d7\" (UniqueName: \"kubernetes.io/projected/68fa4fd4-1346-442b-9781-61233391c11b-kube-api-access-9b7d7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:44.517181 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.517146    4917 generic.go:356] "Generic (PLEG): container finished" podID="68fa4fd4-1346-442b-9781-61233391c11b" containerID="2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d" exitCode=0
Oct 30 20:20:44.517347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.517186    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"
Oct 30 20:20:44.517347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.517228    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" event={"ID":"68fa4fd4-1346-442b-9781-61233391c11b","Type":"ContainerDied","Data":"2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d"}
Oct 30 20:20:44.517347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.517286    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq" event={"ID":"68fa4fd4-1346-442b-9781-61233391c11b","Type":"ContainerDied","Data":"8939a2854b650b1ae505fc6af33d02f0c900293e9b14462c28bef14787f0e667"}
Oct 30 20:20:44.517347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.517308    4917 scope.go:122] "RemoveContainer" containerID="2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d"
Oct 30 20:20:44.527813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.527761    4917 scope.go:122] "RemoveContainer" containerID="2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d"
Oct 30 20:20:44.528101 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:20:44.528079    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d\": container with ID starting with 2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d not found: ID does not exist" containerID="2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d"
Oct 30 20:20:44.528179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.528112    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d"} err="failed to get container status \"2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d\": rpc error: code = NotFound desc = could not find container \"2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d\": container with ID starting with 2efb28df26b4d86a20701af68e80439e06d6a1c01a0f6e8abf378d5dad99cd9d not found: ID does not exist"
Oct 30 20:20:44.690218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.690187    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"]
Oct 30 20:20:44.695145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:44.695119    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k9jsq"]
Oct 30 20:20:46.002338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:46.002306    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fa4fd4-1346-442b-9781-61233391c11b" path="/var/lib/kubelet/pods/68fa4fd4-1346-442b-9781-61233391c11b/volumes"
Oct 30 20:20:59.259012 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.258977    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"]
Oct 30 20:20:59.259530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259487    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy"
Oct 30 20:20:59.259530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259500    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy"
Oct 30 20:20:59.259614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259540    4917 cpu_manager.go:414] "containerMap: removing container" podUID="60e156de-2914-49d7-8a96-dc624914ea68" containerName="revoke-keys"
Oct 30 20:20:59.259614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259545    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e156de-2914-49d7-8a96-dc624914ea68" containerName="revoke-keys"
Oct 30 20:20:59.259614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259554    4917 cpu_manager.go:414] "containerMap: removing container" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy"
Oct 30 20:20:59.259614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259560    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy"
Oct 30 20:20:59.259738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259640    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e156de-2914-49d7-8a96-dc624914ea68" containerName="revoke-keys"
Oct 30 20:20:59.259738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259650    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fa4fd4-1346-442b-9781-61233391c11b" containerName="istio-proxy"
Oct 30 20:20:59.259738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.259660    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ebf533-bdbc-4115-b099-69f34261409c" containerName="istio-proxy"
Oct 30 20:20:59.264677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.264660    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.369210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.369169    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:59.369465 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.369445    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:20:59.369569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.369470    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:59.369631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.369615    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:59.373272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.373252    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"]
Oct 30 20:20:59.425719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.425680    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.425923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.425890    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lg57\" (UniqueName: \"kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.425992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.425971    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.527034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.526927    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.527215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.527052    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lg57\" (UniqueName: \"kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.527215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.527116    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.527863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.527838    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.529508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.529482    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.597602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.597549    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lg57\" (UniqueName: \"kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57\") pod \"maas-api-revoke-sub-ab7256b4-5pflb\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:20:59.879207 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:20:59.879116    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:21:00.271357 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:00.271330    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"]
Oct 30 20:21:00.272634 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:21:00.272612    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1aa486_e16c_4601_a017_2899c7027b79.slice/crio-d0bced4f7070b076b4d815660267dd5be1938ded5d6f559c1439622507c3df4f WatchSource:0}: Error finding container d0bced4f7070b076b4d815660267dd5be1938ded5d6f559c1439622507c3df4f: Status 404 returned error can't find the container with id d0bced4f7070b076b4d815660267dd5be1938ded5d6f559c1439622507c3df4f
Oct 30 20:21:00.662667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:00.662633    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb" event={"ID":"7f1aa486-e16c-4601-a017-2899c7027b79","Type":"ContainerStarted","Data":"6b8c2771d4358710096276bf41126b80020c4ba10648d1771aa89a73c54fb99c"}
Oct 30 20:21:00.662823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:00.662689    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb" event={"ID":"7f1aa486-e16c-4601-a017-2899c7027b79","Type":"ContainerStarted","Data":"d0bced4f7070b076b4d815660267dd5be1938ded5d6f559c1439622507c3df4f"}
Oct 30 20:21:00.811381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:00.811326    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb" podStartSLOduration=1.8113142230000001 podStartE2EDuration="1.811314223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:00.808632173 +0000 UTC m=+2356.973033812" watchObservedRunningTime="2026-10-30 20:21:00.811314223 +0000 UTC m=+2356.975715876"
Oct 30 20:21:01.671759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:01.671673    4917 generic.go:356] "Generic (PLEG): container finished" podID="7f1aa486-e16c-4601-a017-2899c7027b79" containerID="6b8c2771d4358710096276bf41126b80020c4ba10648d1771aa89a73c54fb99c" exitCode=0
Oct 30 20:21:01.671759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:01.671713    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb" event={"ID":"7f1aa486-e16c-4601-a017-2899c7027b79","Type":"ContainerDied","Data":"6b8c2771d4358710096276bf41126b80020c4ba10648d1771aa89a73c54fb99c"}
Oct 30 20:21:01.889215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:01.889174    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"]
Oct 30 20:21:01.892254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:01.892231    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dc56e1-jjl58"]
Oct 30 20:21:02.003496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.003460    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01b566c-858b-407f-82e8-f96ccd137eae" path="/var/lib/kubelet/pods/a01b566c-858b-407f-82e8-f96ccd137eae/volumes"
Oct 30 20:21:02.841285 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.841256    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:21:02.970063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.969978    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca\") pod \"7f1aa486-e16c-4601-a017-2899c7027b79\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") "
Oct 30 20:21:02.970358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.970336    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca\") pod \"7f1aa486-e16c-4601-a017-2899c7027b79\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") "
Oct 30 20:21:02.970643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.970623    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57\" (UniqueName: \"kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57\") pod \"7f1aa486-e16c-4601-a017-2899c7027b79\" (UID: \"7f1aa486-e16c-4601-a017-2899c7027b79\") "
Oct 30 20:21:02.971301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.970652    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca" pod "7f1aa486-e16c-4601-a017-2899c7027b79" (UID: "7f1aa486-e16c-4601-a017-2899c7027b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:02.982989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.982941    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57" pod "7f1aa486-e16c-4601-a017-2899c7027b79" (UID: "7f1aa486-e16c-4601-a017-2899c7027b79"). InnerVolumeSpecName "kube-api-access-2lg57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:02.993900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:02.993851    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca" pod "7f1aa486-e16c-4601-a017-2899c7027b79" (UID: "7f1aa486-e16c-4601-a017-2899c7027b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:03.072347 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:03.072308    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:03.072507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:03.072361    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1aa486-e16c-4601-a017-2899c7027b79-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:03.072507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:03.072377    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lg57\" (UniqueName: \"kubernetes.io/projected/7f1aa486-e16c-4601-a017-2899c7027b79-kube-api-access-2lg57\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:03.687141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:03.687111    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"
Oct 30 20:21:03.687141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:03.687139    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb" event={"ID":"7f1aa486-e16c-4601-a017-2899c7027b79","Type":"ContainerDied","Data":"d0bced4f7070b076b4d815660267dd5be1938ded5d6f559c1439622507c3df4f"}
Oct 30 20:21:03.687141 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:03.687188    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0bced4f7070b076b4d815660267dd5be1938ded5d6f559c1439622507c3df4f"
Oct 30 20:21:09.681312 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:21:09.681265    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:21:19.847366 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.847313    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"]
Oct 30 20:21:19.847885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.847871    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7f1aa486-e16c-4601-a017-2899c7027b79" containerName="revoke-keys"
Oct 30 20:21:19.847955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.847886    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1aa486-e16c-4601-a017-2899c7027b79" containerName="revoke-keys"
Oct 30 20:21:19.848011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.848002    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1aa486-e16c-4601-a017-2899c7027b79" containerName="revoke-keys"
Oct 30 20:21:19.851464 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.851447    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.904636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.904594    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-37732de7-openshift-default-dockercfg-g5kkn\""
Oct 30 20:21:19.943504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943466    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.943665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943579    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.943665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943627    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.943665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943657    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.943800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943733    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.943846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943815    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.943896 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.943880    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.944029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.944012    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqq8n\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.944075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.944045    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:19.959919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:19.959894    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"]
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.045753    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.045841    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.045908    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046042    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqq8n\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046079    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046112    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046169    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046209    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046250    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046827    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.046838    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.047046    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.047313    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.047573    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.049416    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.050217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.049480    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.108832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.108727    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.111693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.111654    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqq8n\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.165606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.165567    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:20.445192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.445158    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"]
Oct 30 20:21:20.448676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.448646    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a2103a-tj2hd"]
Oct 30 20:21:20.614428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.614393    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"]
Oct 30 20:21:20.615182 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:21:20.615158    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23902d2f_c673_49a6_acb5_514f55fa77cb.slice/crio-846b0ca142012b8a13b01513ce81da055d572dcf6f2456282bc8da2cadcd692c WatchSource:0}: Error finding container 846b0ca142012b8a13b01513ce81da055d572dcf6f2456282bc8da2cadcd692c: Status 404 returned error can't find the container with id 846b0ca142012b8a13b01513ce81da055d572dcf6f2456282bc8da2cadcd692c
Oct 30 20:21:20.617839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.617805    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:20.617925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.617873    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:20.617925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.617903    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:20.834252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.833625    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" event={"ID":"23902d2f-c673-49a6-acb5-514f55fa77cb","Type":"ContainerStarted","Data":"24dae2a9567687f48d32a085cc87ecdb66608a899d5d562d25510ff489767602"}
Oct 30 20:21:20.834252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.833687    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" event={"ID":"23902d2f-c673-49a6-acb5-514f55fa77cb","Type":"ContainerStarted","Data":"846b0ca142012b8a13b01513ce81da055d572dcf6f2456282bc8da2cadcd692c"}
Oct 30 20:21:20.964081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:20.963965    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podStartSLOduration=1.963946801 podStartE2EDuration="1.963946801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:20.961828792 +0000 UTC m=+2377.126230434" watchObservedRunningTime="2026-10-30 20:21:20.963946801 +0000 UTC m=+2377.128348442"
Oct 30 20:21:21.165946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:21.165913    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:21.167863 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:21.167836    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:21:21.168000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:21.167887    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 20:21:22.005668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:22.005625    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d956f097-ece8-46e1-ac03-3ddb6e2e0cad" path="/var/lib/kubelet/pods/d956f097-ece8-46e1-ac03-3ddb6e2e0cad/volumes"
Oct 30 20:21:22.167065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:22.167021    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:21:22.167237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:22.167103    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 20:21:23.166256 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:23.166210    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:21:23.166749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:23.166280    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 20:21:24.166253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:24.166162    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:21:24.166415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:24.166258    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 20:21:25.166195 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:25.166154    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:21:25.166412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:25.166213    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 20:21:26.166183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:26.166144    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:21:26.166359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:26.166209    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 20:21:27.170943 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:27.170909    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:27.171344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:27.171325    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:27.172140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:27.172119    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:21:29.570638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.570609    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"]
Oct 30 20:21:29.575181 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.575161    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.618950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.618915    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.619086 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.619010    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.619135 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.619117    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9v2\" (UniqueName: \"kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.630259 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.630231    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:29.634140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.634113    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"]
Oct 30 20:21:29.680496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.680465    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:29.680496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.680474    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:29.680708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.680535    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:21:29.721032 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.720984    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.721669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.721640    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.722836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.722812    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9v2\" (UniqueName: \"kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.723396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.722602    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.748425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.748384    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.789495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.786614    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9v2\" (UniqueName: \"kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2\") pod \"maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:29.889112 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:29.889026    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:30.269156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:30.269124    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"]
Oct 30 20:21:30.269656 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:21:30.269631    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf4b46_3bb8_47fd_9e0c_cef46e509cb8.slice/crio-7f57df6bba76f9cfa0e1befc38c5e26d854ee72ac6cd5f17ebdf42d4596db4c9 WatchSource:0}: Error finding container 7f57df6bba76f9cfa0e1befc38c5e26d854ee72ac6cd5f17ebdf42d4596db4c9: Status 404 returned error can't find the container with id 7f57df6bba76f9cfa0e1befc38c5e26d854ee72ac6cd5f17ebdf42d4596db4c9
Oct 30 20:21:30.963422 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:30.963381    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5" event={"ID":"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8","Type":"ContainerStarted","Data":"d61ac67f070c0c759dcde4446cb87053af6f224ca05781c0f0ae43f3b7fe81cc"}
Oct 30 20:21:30.963990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:30.963461    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5" event={"ID":"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8","Type":"ContainerStarted","Data":"7f57df6bba76f9cfa0e1befc38c5e26d854ee72ac6cd5f17ebdf42d4596db4c9"}
Oct 30 20:21:31.084593 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:31.084538    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5" podStartSLOduration=2.084523392 podStartE2EDuration="2.084523392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:31.081258699 +0000 UTC m=+2387.245660342" watchObservedRunningTime="2026-10-30 20:21:31.084523392 +0000 UTC m=+2387.248925048"
Oct 30 20:21:31.972519 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:31.972487    4917 generic.go:356] "Generic (PLEG): container finished" podID="26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" containerID="d61ac67f070c0c759dcde4446cb87053af6f224ca05781c0f0ae43f3b7fe81cc" exitCode=0
Oct 30 20:21:31.972986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:31.972533    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5" event={"ID":"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8","Type":"ContainerDied","Data":"d61ac67f070c0c759dcde4446cb87053af6f224ca05781c0f0ae43f3b7fe81cc"}
Oct 30 20:21:33.134202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.134173    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:33.269756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.269677    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2\" (UniqueName: \"kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2\") pod \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") "
Oct 30 20:21:33.269756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.269730    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca\") pod \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") "
Oct 30 20:21:33.269956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.269911    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca\") pod \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\" (UID: \"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8\") "
Oct 30 20:21:33.270162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.270138    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca" pod "26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" (UID: "26cf4b46-3bb8-47fd-9e0c-cef46e509cb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:33.270609 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.270583    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.272044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.272018    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca" pod "26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" (UID: "26cf4b46-3bb8-47fd-9e0c-cef46e509cb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:33.272044 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.272035    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2" pod "26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" (UID: "26cf4b46-3bb8-47fd-9e0c-cef46e509cb8"). InnerVolumeSpecName "kube-api-access-rf9v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:33.372348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.372319    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.372348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.372345    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9v2\" (UniqueName: \"kubernetes.io/projected/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8-kube-api-access-rf9v2\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:33.987799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.987731    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5" event={"ID":"26cf4b46-3bb8-47fd-9e0c-cef46e509cb8","Type":"ContainerDied","Data":"7f57df6bba76f9cfa0e1befc38c5e26d854ee72ac6cd5f17ebdf42d4596db4c9"}
Oct 30 20:21:33.987799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.987747    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"
Oct 30 20:21:33.988023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:33.987800    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f57df6bba76f9cfa0e1befc38c5e26d854ee72ac6cd5f17ebdf42d4596db4c9"
Oct 30 20:21:34.247944 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:34.247866    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"]
Oct 30 20:21:34.295929 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:34.295897    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"]
Oct 30 20:21:34.304022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:34.303994    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"]
Oct 30 20:21:35.007523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.007484    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"]
Oct 30 20:21:35.008090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.008074    4917 cpu_manager.go:414] "containerMap: removing container" podUID="26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" containerName="revoke-keys"
Oct 30 20:21:35.008164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.008091    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" containerName="revoke-keys"
Oct 30 20:21:35.008225 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.008186    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" containerName="revoke-keys"
Oct 30 20:21:35.011709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.011693    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.099484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099444    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.099689 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099503    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.099689 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099535    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.099689 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099599    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.099689 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099674    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.100326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099703    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.100326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099919    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.100326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.099954    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx78k\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.100326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.100017    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.103721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.103692    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"]
Oct 30 20:21:35.201014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.200977    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201045    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201067    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201109    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201136    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx78k\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201219    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201281    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201315    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201345    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201461    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201684    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201719    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201790    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.201904 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.201851    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.203804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.203743    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.204092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.204074    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.266088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.266016    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx78k\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.266088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.266015    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.325660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.325627    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:35.758594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.758560    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"]
Oct 30 20:21:35.759382 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:21:35.759356    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ba07f1_cec9_44f4_8e4d_bbf41d5c9f97.slice/crio-29cc96f42b7f5f5976a3b3648e36ec27acccbbbf321c8f3ce7b917d621b2026e WatchSource:0}: Error finding container 29cc96f42b7f5f5976a3b3648e36ec27acccbbbf321c8f3ce7b917d621b2026e: Status 404 returned error can't find the container with id 29cc96f42b7f5f5976a3b3648e36ec27acccbbbf321c8f3ce7b917d621b2026e
Oct 30 20:21:35.761959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.761925    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:35.762036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.761993    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:35.762036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:35.762020    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:36.016106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:36.016012    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" event={"ID":"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97","Type":"ContainerStarted","Data":"ade50926f260e0f473fe812db896be58b06f983f8ceafa7c6c3da527eeb784ff"}
Oct 30 20:21:36.016106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:36.016073    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" event={"ID":"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97","Type":"ContainerStarted","Data":"29cc96f42b7f5f5976a3b3648e36ec27acccbbbf321c8f3ce7b917d621b2026e"}
Oct 30 20:21:36.148885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:36.148830    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podStartSLOduration=2.148816876 podStartE2EDuration="2.148816876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:36.146089453 +0000 UTC m=+2392.310491095" watchObservedRunningTime="2026-10-30 20:21:36.148816876 +0000 UTC m=+2392.313218519"
Oct 30 20:21:36.326276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:36.326195    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:36.327458 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:36.327434    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:36.327575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:36.327487    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:37.326853 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:37.326818    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:37.327242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:37.326878    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:38.326515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:38.326480    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:38.326696 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:38.326535    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:39.039691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.039656    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"]
Oct 30 20:21:39.040202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.039912    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" containerID="cri-o://b89d6a7a0221f8ff47a4c6a654354fbbb9314eb704784be31a47530ff5d1429a" gracePeriod=30
Oct 30 20:21:39.326471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.326389    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:39.326471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.326444    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:39.335006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.334971    4917 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:21:39.335146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.335045    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:21:39.372284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.372243    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"]
Oct 30 20:21:39.376497 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.376470    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.384516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.384490    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.384616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.384595    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.384686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.384665    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8256z\" (UniqueName: \"kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.435258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.435228    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"]
Oct 30 20:21:39.435752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.435724    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:39.435752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.435711    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:39.435966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.435785    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:39.435966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.435877    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:21:39.485621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.485586    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.485848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.485687    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.485848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.485729    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8256z\" (UniqueName: \"kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.486286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.486260    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.488380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.488357    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.570857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.570819    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8256z\" (UniqueName: \"kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z\") pod \"maas-api-revoke-keys-e2e-mt-37732de7-cvhf4\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.691819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.691787    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:39.694457 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:21:39.694412    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:21:39.831578 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.831544    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"]
Oct 30 20:21:39.832116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:39.831871    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" containerID="cri-o://0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8" gracePeriod=30
Oct 30 20:21:40.098360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:40.098321    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"]
Oct 30 20:21:40.098791 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:21:40.098651    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d3db19_f622_4ae8_be72_febce175b77a.slice/crio-f26a4ca3055739c276fe0cc5f45c6ef11a3d1dc8850b18e125617c5f37faa522 WatchSource:0}: Error finding container f26a4ca3055739c276fe0cc5f45c6ef11a3d1dc8850b18e125617c5f37faa522: Status 404 returned error can't find the container with id f26a4ca3055739c276fe0cc5f45c6ef11a3d1dc8850b18e125617c5f37faa522
Oct 30 20:21:40.326069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:40.326035    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:40.326247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:40.326091    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:41.081955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:41.081920    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4" event={"ID":"b1d3db19-f622-4ae8-be72-febce175b77a","Type":"ContainerStarted","Data":"c88c366d8ed5edb5e9203fe62c58b849529a2b50bb2d3ecc8b044b2f3d830357"}
Oct 30 20:21:41.082112 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:41.081981    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4" event={"ID":"b1d3db19-f622-4ae8-be72-febce175b77a","Type":"ContainerStarted","Data":"f26a4ca3055739c276fe0cc5f45c6ef11a3d1dc8850b18e125617c5f37faa522"}
Oct 30 20:21:41.329700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:41.327885    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:41.329700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:41.327948    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:41.521227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:41.521190    4917 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:21:41.521410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:41.521253    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:21:42.092099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.092063    4917 generic.go:356] "Generic (PLEG): container finished" podID="b1d3db19-f622-4ae8-be72-febce175b77a" containerID="c88c366d8ed5edb5e9203fe62c58b849529a2b50bb2d3ecc8b044b2f3d830357" exitCode=0
Oct 30 20:21:42.092322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.092178    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4" event={"ID":"b1d3db19-f622-4ae8-be72-febce175b77a","Type":"ContainerDied","Data":"c88c366d8ed5edb5e9203fe62c58b849529a2b50bb2d3ecc8b044b2f3d830357"}
Oct 30 20:21:42.257832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.257806    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:42.325440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.325404    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca\") pod \"b1d3db19-f622-4ae8-be72-febce175b77a\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") "
Oct 30 20:21:42.325651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.325507    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca\") pod \"b1d3db19-f622-4ae8-be72-febce175b77a\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") "
Oct 30 20:21:42.325745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.325730    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z\" (UniqueName: \"kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z\") pod \"b1d3db19-f622-4ae8-be72-febce175b77a\" (UID: \"b1d3db19-f622-4ae8-be72-febce175b77a\") "
Oct 30 20:21:42.325922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.325894    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca" pod "b1d3db19-f622-4ae8-be72-febce175b77a" (UID: "b1d3db19-f622-4ae8-be72-febce175b77a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:42.326104 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.326079    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:21:42.326170 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.326148    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 20:21:42.326250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.326231    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:42.327680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.327653    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca" pod "b1d3db19-f622-4ae8-be72-febce175b77a" (UID: "b1d3db19-f622-4ae8-be72-febce175b77a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:42.327880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.327858    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z" pod "b1d3db19-f622-4ae8-be72-febce175b77a" (UID: "b1d3db19-f622-4ae8-be72-febce175b77a"). InnerVolumeSpecName "kube-api-access-8256z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:42.427822 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.427739    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d3db19-f622-4ae8-be72-febce175b77a-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:42.427822 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:42.427800    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8256z\" (UniqueName: \"kubernetes.io/projected/b1d3db19-f622-4ae8-be72-febce175b77a-kube-api-access-8256z\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:43.100638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:43.100608    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"
Oct 30 20:21:43.100837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:43.100645    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4" event={"ID":"b1d3db19-f622-4ae8-be72-febce175b77a","Type":"ContainerDied","Data":"f26a4ca3055739c276fe0cc5f45c6ef11a3d1dc8850b18e125617c5f37faa522"}
Oct 30 20:21:43.100837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:43.100693    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f26a4ca3055739c276fe0cc5f45c6ef11a3d1dc8850b18e125617c5f37faa522"
Oct 30 20:21:43.331086 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:43.331054    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:43.331470 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:43.331447    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:43.332072 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:43.332051    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:21:44.094950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.094921    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:21:44.109624 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.109580    4917 generic.go:356] "Generic (PLEG): container finished" podID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerID="b89d6a7a0221f8ff47a4c6a654354fbbb9314eb704784be31a47530ff5d1429a" exitCode=0
Oct 30 20:21:44.109805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.109627    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" event={"ID":"926cf7cf-0c25-4581-9093-08bdeb409afa","Type":"ContainerDied","Data":"b89d6a7a0221f8ff47a4c6a654354fbbb9314eb704784be31a47530ff5d1429a"}
Oct 30 20:21:44.256606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.256574    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:21:44.259727 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259699    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.259872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259795    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.259872 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259857    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.259990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259896    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.259990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259931    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.259990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259966    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.260153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.259994    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.260153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260076    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.260153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260109    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo\") pod \"926cf7cf-0c25-4581-9093-08bdeb409afa\" (UID: \"926cf7cf-0c25-4581-9093-08bdeb409afa\") "
Oct 30 20:21:44.260312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260129    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:44.260312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260178    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:44.260312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260298    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:44.260423 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260382    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:44.260463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260416    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:44.260695 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260674    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.260695 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260693    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926cf7cf-0c25-4581-9093-08bdeb409afa-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.260908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260708    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.260908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260718    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.260908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.260728    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.262453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.262426    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "kube-api-access-d6zhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:44.262453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.262434    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:44.262579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.262463    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:21:44.262579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.262569    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token" pod "926cf7cf-0c25-4581-9093-08bdeb409afa" (UID: "926cf7cf-0c25-4581-9093-08bdeb409afa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:44.340499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.340423    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:21:44.361604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.361565    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.361604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.361594    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.361604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.361608    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926cf7cf-0c25-4581-9093-08bdeb409afa-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.361939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.361621    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zhx\" (UniqueName: \"kubernetes.io/projected/926cf7cf-0c25-4581-9093-08bdeb409afa-kube-api-access-d6zhx\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:44.391284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:44.391261    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:21:45.115198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.115137    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:21:45.117837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.117809    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw" event={"ID":"926cf7cf-0c25-4581-9093-08bdeb409afa","Type":"ContainerDied","Data":"12f7dc82d95227e65bee3224a44579118c4da7c11c7dbdcc8a628dc45ee74a79"}
Oct 30 20:21:45.117960 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.117841    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"
Oct 30 20:21:45.117960 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.117882    4917 scope.go:122] "RemoveContainer" containerID="b89d6a7a0221f8ff47a4c6a654354fbbb9314eb704784be31a47530ff5d1429a"
Oct 30 20:21:45.119345 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.119319    4917 generic.go:356] "Generic (PLEG): container finished" podID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerID="0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8" exitCode=0
Oct 30 20:21:45.119449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.119359    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"
Oct 30 20:21:45.119449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.119367    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" event={"ID":"36dc42f7-4cce-47b5-b6c2-c37577cce641","Type":"ContainerDied","Data":"0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8"}
Oct 30 20:21:45.119449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.119416    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w" event={"ID":"36dc42f7-4cce-47b5-b6c2-c37577cce641","Type":"ContainerDied","Data":"851eb47ccbb1ad5ff4aaf5e10a13fc97215ba83ca9b1ad001773a38311224321"}
Oct 30 20:21:45.132212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.132194    4917 scope.go:122] "RemoveContainer" containerID="0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8"
Oct 30 20:21:45.142126 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.142103    4917 scope.go:122] "RemoveContainer" containerID="0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8"
Oct 30 20:21:45.142392 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:21:45.142360    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8\": container with ID starting with 0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8 not found: ID does not exist" containerID="0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8"
Oct 30 20:21:45.142474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.142397    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8"} err="failed to get container status \"0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8\": rpc error: code = NotFound desc = could not find container \"0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8\": container with ID starting with 0a0c76427979a3e04da744f11673bc5caa8024ad156490a1a72b9c1db3b552c8 not found: ID does not exist"
Oct 30 20:21:45.170525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170499    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.170665 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170606    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.170732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170679    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.170732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170717    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.170905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170831    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.170905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170896    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.171013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.170960    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.171066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171023    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.171066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171055    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs\") pod \"36dc42f7-4cce-47b5-b6c2-c37577cce641\" (UID: \"36dc42f7-4cce-47b5-b6c2-c37577cce641\") "
Oct 30 20:21:45.171164 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171133    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.171397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171375    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.171472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171387    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:45.171669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171646    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.171785 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171749    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.171861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171791    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc42f7-4cce-47b5-b6c2-c37577cce641-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.171861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171807    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.171861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171820    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.171861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.171853    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.173112 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.173086    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.173222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.173124    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:45.173490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.173468    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:21:45.173667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.173651    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd" pod "36dc42f7-4cce-47b5-b6c2-c37577cce641" (UID: "36dc42f7-4cce-47b5-b6c2-c37577cce641"). InnerVolumeSpecName "kube-api-access-8cdfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:45.273122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.273092    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.273122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.273119    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.273122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.273129    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.273346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.273138    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cdfd\" (UniqueName: \"kubernetes.io/projected/36dc42f7-4cce-47b5-b6c2-c37577cce641-kube-api-access-8cdfd\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.273346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.273150    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc42f7-4cce-47b5-b6c2-c37577cce641-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.437482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.437447    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"]
Oct 30 20:21:45.443836 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.442736    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-bnkzw"]
Oct 30 20:21:45.461147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.461124    4917 scope.go:122] "RemoveContainer" containerID="966103cf70948c5d244d05dafd191e72b34abdc6e5cf80caa4d31590d2f283e0"
Oct 30 20:21:45.604066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.604031    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"]
Oct 30 20:21:45.611709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:45.611675    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-95m4w"]
Oct 30 20:21:46.004006 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:46.003974    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" path="/var/lib/kubelet/pods/36dc42f7-4cce-47b5-b6c2-c37577cce641/volumes"
Oct 30 20:21:46.004483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:46.004469    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" path="/var/lib/kubelet/pods/926cf7cf-0c25-4581-9093-08bdeb409afa/volumes"
Oct 30 20:21:49.268466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.268429    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main" containerID="cri-o://5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c" gracePeriod=45
Oct 30 20:21:49.317272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.317223    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main" containerID="cri-o://ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63" gracePeriod=45
Oct 30 20:21:49.324373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.324339    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main" containerID="cri-o://6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515" gracePeriod=45
Oct 30 20:21:49.617143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.616831    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:21:49.629166 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.629093    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:21:49.649489 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.649464    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:21:49.738113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738035    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir\") pod \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") "
Oct 30 20:21:49.738320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738197    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir\") pod \"77350f60-57fa-40a5-8fe5-742c5b5f7979\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") "
Oct 30 20:21:49.738320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738241    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir\") pod \"06528167-b423-4021-a6da-6fd10e1eab57\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") "
Oct 30 20:21:49.738320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738301    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs\" (UniqueName: \"kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs\") pod \"77350f60-57fa-40a5-8fe5-742c5b5f7979\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") "
Oct 30 20:21:49.738502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738346    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7\" (UniqueName: \"kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7\") pod \"77350f60-57fa-40a5-8fe5-742c5b5f7979\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") "
Oct 30 20:21:49.738502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738424    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d\" (UniqueName: \"kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d\") pod \"06528167-b423-4021-a6da-6fd10e1eab57\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") "
Oct 30 20:21:49.738502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738472    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm\") pod \"77350f60-57fa-40a5-8fe5-742c5b5f7979\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") "
Oct 30 20:21:49.738666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738505    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home\") pod \"06528167-b423-4021-a6da-6fd10e1eab57\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") "
Oct 30 20:21:49.738666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738531    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm\") pod \"06528167-b423-4021-a6da-6fd10e1eab57\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") "
Oct 30 20:21:49.738666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738587    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home\") pod \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") "
Oct 30 20:21:49.738666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738646    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs\") pod \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") "
Oct 30 20:21:49.738923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738694    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache\") pod \"77350f60-57fa-40a5-8fe5-742c5b5f7979\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") "
Oct 30 20:21:49.738923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738744    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp\" (UniqueName: \"kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp\") pod \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") "
Oct 30 20:21:49.738923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738802    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache\") pod \"06528167-b423-4021-a6da-6fd10e1eab57\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") "
Oct 30 20:21:49.738923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738896    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home\") pod \"77350f60-57fa-40a5-8fe5-742c5b5f7979\" (UID: \"77350f60-57fa-40a5-8fe5-742c5b5f7979\") "
Oct 30 20:21:49.739154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.738997    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm\") pod \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") "
Oct 30 20:21:49.739154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.739030    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache\") pod \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\" (UID: \"a88a932e-0217-4f8f-a956-54e5cef8e7b1\") "
Oct 30 20:21:49.739570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.739545    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home" pod "06528167-b423-4021-a6da-6fd10e1eab57" (UID: "06528167-b423-4021-a6da-6fd10e1eab57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.739913 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.739888    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir" pod "a88a932e-0217-4f8f-a956-54e5cef8e7b1" (UID: "a88a932e-0217-4f8f-a956-54e5cef8e7b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.740094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.740077    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir" pod "77350f60-57fa-40a5-8fe5-742c5b5f7979" (UID: "77350f60-57fa-40a5-8fe5-742c5b5f7979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.740270 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.740247    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir" pod "06528167-b423-4021-a6da-6fd10e1eab57" (UID: "06528167-b423-4021-a6da-6fd10e1eab57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.741541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.741137    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache" pod "77350f60-57fa-40a5-8fe5-742c5b5f7979" (UID: "77350f60-57fa-40a5-8fe5-742c5b5f7979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.741541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.741245    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache" pod "06528167-b423-4021-a6da-6fd10e1eab57" (UID: "06528167-b423-4021-a6da-6fd10e1eab57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.741541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.741452    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs\" (UniqueName: \"kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs\") pod \"06528167-b423-4021-a6da-6fd10e1eab57\" (UID: \"06528167-b423-4021-a6da-6fd10e1eab57\") "
Oct 30 20:21:49.742083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.742063    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.742083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.742090    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.742212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.742104    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.742212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.742141    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.742212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.742157    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.742212 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.742170    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.745430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.744889    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache" pod "a88a932e-0217-4f8f-a956-54e5cef8e7b1" (UID: "a88a932e-0217-4f8f-a956-54e5cef8e7b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.745430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.745070    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home" pod "77350f60-57fa-40a5-8fe5-742c5b5f7979" (UID: "77350f60-57fa-40a5-8fe5-742c5b5f7979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.745430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.745215    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home" pod "a88a932e-0217-4f8f-a956-54e5cef8e7b1" (UID: "a88a932e-0217-4f8f-a956-54e5cef8e7b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.747346 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.747305    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs" pod "77350f60-57fa-40a5-8fe5-742c5b5f7979" (UID: "77350f60-57fa-40a5-8fe5-742c5b5f7979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:49.751186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751129    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs" pod "06528167-b423-4021-a6da-6fd10e1eab57" (UID: "06528167-b423-4021-a6da-6fd10e1eab57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:49.751186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751140    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm" pod "77350f60-57fa-40a5-8fe5-742c5b5f7979" (UID: "77350f60-57fa-40a5-8fe5-742c5b5f7979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.751186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751155    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp" pod "a88a932e-0217-4f8f-a956-54e5cef8e7b1" (UID: "a88a932e-0217-4f8f-a956-54e5cef8e7b1"). InnerVolumeSpecName "kube-api-access-2mgtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:49.751526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751486    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm" pod "06528167-b423-4021-a6da-6fd10e1eab57" (UID: "06528167-b423-4021-a6da-6fd10e1eab57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.751526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751492    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7" pod "77350f60-57fa-40a5-8fe5-742c5b5f7979" (UID: "77350f60-57fa-40a5-8fe5-742c5b5f7979"). InnerVolumeSpecName "kube-api-access-z86v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:49.751699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751524    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d" pod "06528167-b423-4021-a6da-6fd10e1eab57" (UID: "06528167-b423-4021-a6da-6fd10e1eab57"). InnerVolumeSpecName "kube-api-access-j7x8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:49.751984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.751958    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm" pod "a88a932e-0217-4f8f-a956-54e5cef8e7b1" (UID: "a88a932e-0217-4f8f-a956-54e5cef8e7b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.752928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.752903    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs" pod "a88a932e-0217-4f8f-a956-54e5cef8e7b1" (UID: "a88a932e-0217-4f8f-a956-54e5cef8e7b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843636    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77350f60-57fa-40a5-8fe5-742c5b5f7979-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843664    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z86v7\" (UniqueName: \"kubernetes.io/projected/77350f60-57fa-40a5-8fe5-742c5b5f7979-kube-api-access-z86v7\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843674    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7x8d\" (UniqueName: \"kubernetes.io/projected/06528167-b423-4021-a6da-6fd10e1eab57-kube-api-access-j7x8d\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843682    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843691    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06528167-b423-4021-a6da-6fd10e1eab57-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843699    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843707    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a932e-0217-4f8f-a956-54e5cef8e7b1-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.843712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843715    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mgtp\" (UniqueName: \"kubernetes.io/projected/a88a932e-0217-4f8f-a956-54e5cef8e7b1-kube-api-access-2mgtp\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.844132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843723    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77350f60-57fa-40a5-8fe5-742c5b5f7979-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.844132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843732    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.844132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843740    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a932e-0217-4f8f-a956-54e5cef8e7b1-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.844132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:49.843749    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06528167-b423-4021-a6da-6fd10e1eab57-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:50.128227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128129    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"]
Oct 30 20:21:50.128670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128654    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128669    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128689    4917 cpu_manager.go:414] "containerMap: removing container" podUID="b1d3db19-f622-4ae8-be72-febce175b77a" containerName="revoke-keys"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128694    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d3db19-f622-4ae8-be72-febce175b77a" containerName="revoke-keys"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128704    4917 cpu_manager.go:414] "containerMap: removing container" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128709    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128718    4917 cpu_manager.go:414] "containerMap: removing container" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128723    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128738    4917 cpu_manager.go:414] "containerMap: removing container" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128743    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128751    4917 cpu_manager.go:414] "containerMap: removing container" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy"
Oct 30 20:21:50.128753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128756    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy"
Oct 30 20:21:50.129334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128864    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerName="main"
Oct 30 20:21:50.129334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128875    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d3db19-f622-4ae8-be72-febce175b77a" containerName="revoke-keys"
Oct 30 20:21:50.129334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128882    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerName="main"
Oct 30 20:21:50.129334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128888    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dc42f7-4cce-47b5-b6c2-c37577cce641" containerName="istio-proxy"
Oct 30 20:21:50.129334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128895    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="06528167-b423-4021-a6da-6fd10e1eab57" containerName="main"
Oct 30 20:21:50.129334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.128901    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="926cf7cf-0c25-4581-9093-08bdeb409afa" containerName="istio-proxy"
Oct 30 20:21:50.133627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.133596    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.157933 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.157900    4917 generic.go:356] "Generic (PLEG): container finished" podID="77350f60-57fa-40a5-8fe5-742c5b5f7979" containerID="6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515" exitCode=0
Oct 30 20:21:50.158094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.157941    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"
Oct 30 20:21:50.158094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.157980    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" event={"ID":"77350f60-57fa-40a5-8fe5-742c5b5f7979","Type":"ContainerDied","Data":"6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515"}
Oct 30 20:21:50.158094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.158023    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl" event={"ID":"77350f60-57fa-40a5-8fe5-742c5b5f7979","Type":"ContainerDied","Data":"b9dd41fd10f5904da06725a0d4edc9a21e3724e4f036a1780c5de508ea4f7633"}
Oct 30 20:21:50.158094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.158046    4917 scope.go:122] "RemoveContainer" containerID="6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515"
Oct 30 20:21:50.159549 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.159531    4917 generic.go:356] "Generic (PLEG): container finished" podID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" containerID="ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63" exitCode=0
Oct 30 20:21:50.159622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.159590    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"
Oct 30 20:21:50.159678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.159615    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" event={"ID":"a88a932e-0217-4f8f-a956-54e5cef8e7b1","Type":"ContainerDied","Data":"ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63"}
Oct 30 20:21:50.159678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.159665    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz" event={"ID":"a88a932e-0217-4f8f-a956-54e5cef8e7b1","Type":"ContainerDied","Data":"478b55165f7266edb7fcc1290861751049be43ca4c494a6d1a73648fd5813061"}
Oct 30 20:21:50.161552 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.161522    4917 generic.go:356] "Generic (PLEG): container finished" podID="06528167-b423-4021-a6da-6fd10e1eab57" containerID="5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c" exitCode=0
Oct 30 20:21:50.161666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.161592    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" event={"ID":"06528167-b423-4021-a6da-6fd10e1eab57","Type":"ContainerDied","Data":"5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c"}
Oct 30 20:21:50.161666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.161622    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh" event={"ID":"06528167-b423-4021-a6da-6fd10e1eab57","Type":"ContainerDied","Data":"2ea68c701fc09c09af025fbcb2c01d91c9ad1782e802b5a65297371ac3cdb623"}
Oct 30 20:21:50.161801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.161678    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"
Oct 30 20:21:50.170034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.170015    4917 scope.go:122] "RemoveContainer" containerID="6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515"
Oct 30 20:21:50.170305 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:21:50.170279    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515\": container with ID starting with 6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515 not found: ID does not exist" containerID="6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515"
Oct 30 20:21:50.170397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.170315    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515"} err="failed to get container status \"6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515\": rpc error: code = NotFound desc = could not find container \"6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515\": container with ID starting with 6386cc9b1a6fa7d2470d29c225730c46dc28d10d729645ccff5467a5f393f515 not found: ID does not exist"
Oct 30 20:21:50.170397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.170342    4917 scope.go:122] "RemoveContainer" containerID="ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63"
Oct 30 20:21:50.179707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.179689    4917 scope.go:122] "RemoveContainer" containerID="ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63"
Oct 30 20:21:50.179962 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:21:50.179943    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63\": container with ID starting with ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63 not found: ID does not exist" containerID="ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63"
Oct 30 20:21:50.180017 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.179969    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63"} err="failed to get container status \"ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63\": rpc error: code = NotFound desc = could not find container \"ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63\": container with ID starting with ee087f3df3c8d6b363f2465138b52d6238dc3eaaadf18d92d588860704073c63 not found: ID does not exist"
Oct 30 20:21:50.180017 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.179989    4917 scope.go:122] "RemoveContainer" containerID="5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c"
Oct 30 20:21:50.193413 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.192950    4917 scope.go:122] "RemoveContainer" containerID="5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c"
Oct 30 20:21:50.194042 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:21:50.193471    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c\": container with ID starting with 5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c not found: ID does not exist" containerID="5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c"
Oct 30 20:21:50.194400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.194283    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c"} err="failed to get container status \"5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c\": rpc error: code = NotFound desc = could not find container \"5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c\": container with ID starting with 5e587506f1811035f7cafac3d34f9b60f8a5122026d5c26dad8d82224565002c not found: ID does not exist"
Oct 30 20:21:50.198140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.198119    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"]
Oct 30 20:21:50.248734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248699    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.248935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248749    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.248935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248814    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.248935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248872    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.248935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248902    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.248935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248929    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.249129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.248966    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.249129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.249045    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fr2\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.249129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.249108    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350308    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350360    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fr2\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350423    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350523    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350568    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350600    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.350765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350690    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.350806    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.351005    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.351012    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.351118    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351196 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.351155    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351536 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.351513    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.351754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.351738    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.353188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.353161    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.353295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.353188    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.374145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.374120    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"]
Oct 30 20:21:50.381048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.380984    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/simulator-w2-d67ea9-kserve-5dd6cd6649-sr6bl"]
Oct 30 20:21:50.412697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.412675    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.413095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.413077    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fr2\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.451457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.451433    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:50.492941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.492903    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"]
Oct 30 20:21:50.509201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.509157    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/premium-w2-d67ea9-kserve-7cdbd79bfb-rjnbz"]
Oct 30 20:21:50.612800 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.612756    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"]
Oct 30 20:21:50.623150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.623119    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-2-w2-d67ea9-kserve-6cdd67c4c6-6bxdh"]
Oct 30 20:21:50.891087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.891057    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"]
Oct 30 20:21:50.892081 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:21:50.892046    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeb3d71_c8f2_408f_b54a_773d051fdb23.slice/crio-d367cf98629b48da60e08ed676ab98128ebed5d0048a71ed7ff9396b37b9de2b WatchSource:0}: Error finding container d367cf98629b48da60e08ed676ab98128ebed5d0048a71ed7ff9396b37b9de2b: Status 404 returned error can't find the container with id d367cf98629b48da60e08ed676ab98128ebed5d0048a71ed7ff9396b37b9de2b
Oct 30 20:21:50.894490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.894458    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:50.894565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.894524    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:50.894565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:50.894553    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:21:51.173043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:51.172950    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" event={"ID":"fbeb3d71-c8f2-408f-b54a-773d051fdb23","Type":"ContainerStarted","Data":"1a65ad88ee127938c7b4bb2804a4232989e95d6ef3eba47bbf2a27ae07616f8f"}
Oct 30 20:21:51.173043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:51.173003    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" event={"ID":"fbeb3d71-c8f2-408f-b54a-773d051fdb23","Type":"ContainerStarted","Data":"d367cf98629b48da60e08ed676ab98128ebed5d0048a71ed7ff9396b37b9de2b"}
Oct 30 20:21:51.296123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:51.296054    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podStartSLOduration=1.296034592 podStartE2EDuration="1.296034592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:51.294959131 +0000 UTC m=+2407.459360774" watchObservedRunningTime="2026-10-30 20:21:51.296034592 +0000 UTC m=+2407.460436237"
Oct 30 20:21:51.451841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:51.451810    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:51.453719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:51.453695    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:21:51.453849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:51.453749    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 20:21:52.003903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:52.003871    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06528167-b423-4021-a6da-6fd10e1eab57" path="/var/lib/kubelet/pods/06528167-b423-4021-a6da-6fd10e1eab57/volumes"
Oct 30 20:21:52.004365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:52.004351    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77350f60-57fa-40a5-8fe5-742c5b5f7979" path="/var/lib/kubelet/pods/77350f60-57fa-40a5-8fe5-742c5b5f7979/volumes"
Oct 30 20:21:52.004753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:52.004741    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88a932e-0217-4f8f-a956-54e5cef8e7b1" path="/var/lib/kubelet/pods/a88a932e-0217-4f8f-a956-54e5cef8e7b1/volumes"
Oct 30 20:21:52.458721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:52.458235    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:21:52.458721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:52.458303    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 20:21:53.465799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:53.462921    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:21:53.465799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:53.463008    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 20:21:54.453128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:54.453085    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:21:54.453331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:54.453150    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 20:21:55.475799 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:55.469267    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:21:55.480095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:55.480051    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 20:21:56.452620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:56.452157    4917 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:21:56.452620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:56.452221    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 20:21:58.092051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:58.092018    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:58.092602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:58.092576    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:58.093660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:58.093636    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:21:58.218898 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:58.218865    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"]
Oct 30 20:21:58.222449 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:21:58.222426    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f84d85-w4gcc"]
Oct 30 20:22:00.002510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:00.002478    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e6142d-ad13-4cb3-be7e-5024e4aec2db" path="/var/lib/kubelet/pods/57e6142d-ad13-4cb3-be7e-5024e4aec2db/volumes"
Oct 30 20:22:07.485859 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.485818    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"]
Oct 30 20:22:07.490099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.490073    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.532415 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.532331    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.532632 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.532491    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2mg\" (UniqueName: \"kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.532632 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.532576    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.543743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.543710    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:07.594039 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.594006    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:22:07.594248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.594105    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:22:07.594248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.594177    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:22:07.597788 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.597748    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"]
Oct 30 20:22:07.634294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.634259    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2mg\" (UniqueName: \"kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.634450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.634300    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.634450 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.634389    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.635011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.634986    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.636928 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.636905    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.696670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.696637    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2mg\" (UniqueName: \"kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg\") pod \"maas-api-revoke-sub-1cd054f2-gsnxp\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:07.804982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:07.804892    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:08.190054 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:08.190012    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"]
Oct 30 20:22:08.199740 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:22:08.199685    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd32c29_63ce_4952_8e4d_e9b4a909e7eb.slice/crio-eace06e88daf6173122b7f409665fc8f1e9b818796a9881a5edfb2f13bc26be0 WatchSource:0}: Error finding container eace06e88daf6173122b7f409665fc8f1e9b818796a9881a5edfb2f13bc26be0: Status 404 returned error can't find the container with id eace06e88daf6173122b7f409665fc8f1e9b818796a9881a5edfb2f13bc26be0
Oct 30 20:22:08.327687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:08.327640    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp" event={"ID":"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb","Type":"ContainerStarted","Data":"eace06e88daf6173122b7f409665fc8f1e9b818796a9881a5edfb2f13bc26be0"}
Oct 30 20:22:09.337427 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:09.337394    4917 generic.go:356] "Generic (PLEG): container finished" podID="6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" containerID="52bb9297763e5b72d48d86fd496cc81ba7d4b70892ed7ec479a23d0d22ac29bc" exitCode=0
Oct 30 20:22:09.337974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:09.337478    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp" event={"ID":"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb","Type":"ContainerDied","Data":"52bb9297763e5b72d48d86fd496cc81ba7d4b70892ed7ec479a23d0d22ac29bc"}
Oct 30 20:22:09.683913 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:22:09.683866    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:22:10.515180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.515151    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:10.578878 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.578840    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca\") pod \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") "
Oct 30 20:22:10.579117 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.579097    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg\" (UniqueName: \"kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg\") pod \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") "
Oct 30 20:22:10.579199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.579142    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca\") pod \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\" (UID: \"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb\") "
Oct 30 20:22:10.579546 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.579515    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca" pod "6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" (UID: "6cd32c29-63ce-4952-8e4d-e9b4a909e7eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:10.581418 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.581394    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg" pod "6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" (UID: "6cd32c29-63ce-4952-8e4d-e9b4a909e7eb"). InnerVolumeSpecName "kube-api-access-lv2mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:10.581587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.581565    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca" pod "6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" (UID: "6cd32c29-63ce-4952-8e4d-e9b4a909e7eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:10.680916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.680880    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:10.680916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.680914    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv2mg\" (UniqueName: \"kubernetes.io/projected/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-kube-api-access-lv2mg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:10.681128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:10.680929    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.355683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:11.355640    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp" event={"ID":"6cd32c29-63ce-4952-8e4d-e9b4a909e7eb","Type":"ContainerDied","Data":"eace06e88daf6173122b7f409665fc8f1e9b818796a9881a5edfb2f13bc26be0"}
Oct 30 20:22:11.355946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:11.355691    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"
Oct 30 20:22:11.355946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:11.355708    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eace06e88daf6173122b7f409665fc8f1e9b818796a9881a5edfb2f13bc26be0"
Oct 30 20:22:20.924755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:20.924720    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"]
Oct 30 20:22:20.975276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:20.975234    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37732de7-cvhf4"]
Oct 30 20:22:22.003250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:22.003215    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d3db19-f622-4ae8-be72-febce175b77a" path="/var/lib/kubelet/pods/b1d3db19-f622-4ae8-be72-febce175b77a/volumes"
Oct 30 20:22:26.319840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:26.319796    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"]
Oct 30 20:22:30.800288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:30.800243    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:22:39.680912 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:22:39.680873    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:22:41.341123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.341080    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" podUID="be8fcdc8-23a3-4899-8351-56de8211847f" containerName="main" containerID="cri-o://0af1d2e5ff639399b77154666771b182f0b877c2ec0fb4bed93fe8f41be3a016" gracePeriod=45
Oct 30 20:22:41.445986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.445941    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" podUID="be8fcdc8-23a3-4899-8351-56de8211847f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": dial tcp 10.132.0.154:8000: connect: connection refused"
Oct 30 20:22:41.594263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.594238    4917 generic.go:356] "Generic (PLEG): container finished" podID="be8fcdc8-23a3-4899-8351-56de8211847f" containerID="0af1d2e5ff639399b77154666771b182f0b877c2ec0fb4bed93fe8f41be3a016" exitCode=0
Oct 30 20:22:41.594388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.594289    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" event={"ID":"be8fcdc8-23a3-4899-8351-56de8211847f","Type":"ContainerDied","Data":"0af1d2e5ff639399b77154666771b182f0b877c2ec0fb4bed93fe8f41be3a016"}
Oct 30 20:22:41.594388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.594326    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h" event={"ID":"be8fcdc8-23a3-4899-8351-56de8211847f","Type":"ContainerDied","Data":"6064b7861e71c629574f8f512955f4f72bcf090535bbd11236a240958a35a9cc"}
Oct 30 20:22:41.594388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.594343    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6064b7861e71c629574f8f512955f4f72bcf090535bbd11236a240958a35a9cc"
Oct 30 20:22:41.612778 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.612738    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:22:41.719668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719632    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache\") pod \"be8fcdc8-23a3-4899-8351-56de8211847f\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") "
Oct 30 20:22:41.719890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719675    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home\") pod \"be8fcdc8-23a3-4899-8351-56de8211847f\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") "
Oct 30 20:22:41.719890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719705    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm\") pod \"be8fcdc8-23a3-4899-8351-56de8211847f\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") "
Oct 30 20:22:41.719890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719754    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs\" (UniqueName: \"kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs\") pod \"be8fcdc8-23a3-4899-8351-56de8211847f\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") "
Oct 30 20:22:41.719890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719834    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk\" (UniqueName: \"kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk\") pod \"be8fcdc8-23a3-4899-8351-56de8211847f\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") "
Oct 30 20:22:41.719890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719868    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir\") pod \"be8fcdc8-23a3-4899-8351-56de8211847f\" (UID: \"be8fcdc8-23a3-4899-8351-56de8211847f\") "
Oct 30 20:22:41.720188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.719975    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache" pod "be8fcdc8-23a3-4899-8351-56de8211847f" (UID: "be8fcdc8-23a3-4899-8351-56de8211847f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:41.720188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.720077    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home" pod "be8fcdc8-23a3-4899-8351-56de8211847f" (UID: "be8fcdc8-23a3-4899-8351-56de8211847f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:41.720188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.720158    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir" pod "be8fcdc8-23a3-4899-8351-56de8211847f" (UID: "be8fcdc8-23a3-4899-8351-56de8211847f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:41.720505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.720444    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:41.720505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.720464    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:41.720505 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.720474    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:41.722219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.722190    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs" pod "be8fcdc8-23a3-4899-8351-56de8211847f" (UID: "be8fcdc8-23a3-4899-8351-56de8211847f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:41.722330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.722191    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm" pod "be8fcdc8-23a3-4899-8351-56de8211847f" (UID: "be8fcdc8-23a3-4899-8351-56de8211847f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:41.722330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.722286    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk" pod "be8fcdc8-23a3-4899-8351-56de8211847f" (UID: "be8fcdc8-23a3-4899-8351-56de8211847f"). InnerVolumeSpecName "kube-api-access-znllk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:41.821575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.821537    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8fcdc8-23a3-4899-8351-56de8211847f-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:41.821575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.821571    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8fcdc8-23a3-4899-8351-56de8211847f-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:41.821816 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:41.821588    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znllk\" (UniqueName: \"kubernetes.io/projected/be8fcdc8-23a3-4899-8351-56de8211847f-kube-api-access-znllk\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:42.601720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:42.601682    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"
Oct 30 20:22:42.771989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:42.771952    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"]
Oct 30 20:22:42.822055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:42.822020    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-37732de7/e2e-lifecycle-model-37732de7-kserve-77fb86d78f-cw98h"]
Oct 30 20:22:44.003637 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:44.003603    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8fcdc8-23a3-4899-8351-56de8211847f" path="/var/lib/kubelet/pods/be8fcdc8-23a3-4899-8351-56de8211847f/volumes"
Oct 30 20:22:45.497744 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:22:45.497712    4917 scope.go:122] "RemoveContainer" containerID="0af1d2e5ff639399b77154666771b182f0b877c2ec0fb4bed93fe8f41be3a016"
Oct 30 20:23:04.054542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:04.054500    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"]
Oct 30 20:23:04.103311 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:04.103273    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f115d539-9qgdm"]
Oct 30 20:23:06.003330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:06.003291    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81f1fe9-9614-42c6-9d20-0b57ccf851b3" path="/var/lib/kubelet/pods/e81f1fe9-9614-42c6-9d20-0b57ccf851b3/volumes"
Oct 30 20:23:09.680675 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:23:09.680636    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:23:09.724707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:09.724672    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"]
Oct 30 20:23:09.775055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:09.775025    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d67ea9-c97n5"]
Oct 30 20:23:10.003193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:10.003116    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cf4b46-3bb8-47fd-9e0c-cef46e509cb8" path="/var/lib/kubelet/pods/26cf4b46-3bb8-47fd-9e0c-cef46e509cb8/volumes"
Oct 30 20:23:14.860996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.860941    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"]
Oct 30 20:23:14.861616 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.861598    4917 cpu_manager.go:414] "containerMap: removing container" podUID="be8fcdc8-23a3-4899-8351-56de8211847f" containerName="main"
Oct 30 20:23:14.861682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.861616    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8fcdc8-23a3-4899-8351-56de8211847f" containerName="main"
Oct 30 20:23:14.861682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.861647    4917 cpu_manager.go:414] "containerMap: removing container" podUID="6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" containerName="revoke-keys"
Oct 30 20:23:14.861682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.861654    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" containerName="revoke-keys"
Oct 30 20:23:14.861821 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.861750    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" containerName="revoke-keys"
Oct 30 20:23:14.861821 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.861783    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8fcdc8-23a3-4899-8351-56de8211847f" containerName="main"
Oct 30 20:23:14.867111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.867089    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:14.924123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.924088    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"]
Oct 30 20:23:14.964448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.964417    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj69\" (UniqueName: \"kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:14.964626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.964461    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:14.964626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.964605    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:14.970143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.970120    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:14.970524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.970450    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:14.970524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.970477    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:23:14.970658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:14.970557    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:15.065998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.065967    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.066183 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.066162    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj69\" (UniqueName: \"kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.066228 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.066202    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.066893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.066872    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.068480 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.068457    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.127752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.127682    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj69\" (UniqueName: \"kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.180980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.180947    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:15.558995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.558969    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"]
Oct 30 20:23:15.559563 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:23:15.559534    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143d16d4_48d0_4257_9d59_3f475da2c68d.slice/crio-b8bee2a712b5056b99e867adfb11e8db058fed2ed12205ce4fa594ebc31ed88b WatchSource:0}: Error finding container b8bee2a712b5056b99e867adfb11e8db058fed2ed12205ce4fa594ebc31ed88b: Status 404 returned error can't find the container with id b8bee2a712b5056b99e867adfb11e8db058fed2ed12205ce4fa594ebc31ed88b
Oct 30 20:23:15.866199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.866171    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp" event={"ID":"143d16d4-48d0-4257-9d59-3f475da2c68d","Type":"ContainerStarted","Data":"97b1f4a610560d9546abc2c9a17885aff120294dc691c26e429ab9934909e14b"}
Oct 30 20:23:15.866523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.866219    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp" event={"ID":"143d16d4-48d0-4257-9d59-3f475da2c68d","Type":"ContainerStarted","Data":"b8bee2a712b5056b99e867adfb11e8db058fed2ed12205ce4fa594ebc31ed88b"}
Oct 30 20:23:15.985847 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:15.985797    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp" podStartSLOduration=1.9857828579999999 podStartE2EDuration="1.985782858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:15.983661127 +0000 UTC m=+2492.148062770" watchObservedRunningTime="2026-10-30 20:23:15.985782858 +0000 UTC m=+2492.150184490"
Oct 30 20:23:16.876156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:16.876117    4917 generic.go:356] "Generic (PLEG): container finished" podID="143d16d4-48d0-4257-9d59-3f475da2c68d" containerID="97b1f4a610560d9546abc2c9a17885aff120294dc691c26e429ab9934909e14b" exitCode=0
Oct 30 20:23:16.876570 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:16.876201    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp" event={"ID":"143d16d4-48d0-4257-9d59-3f475da2c68d","Type":"ContainerDied","Data":"97b1f4a610560d9546abc2c9a17885aff120294dc691c26e429ab9934909e14b"}
Oct 30 20:23:17.807809 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:17.807758    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"]
Oct 30 20:23:17.808084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:17.808058    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy" containerID="cri-o://35c8d37fb7ce36612f5f84189a3ef31b61d864de14cf6d9592f6bacf5bfd8872" gracePeriod=30
Oct 30 20:23:18.123813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.123759    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:18.220814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.220743    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca\") pod \"143d16d4-48d0-4257-9d59-3f475da2c68d\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") "
Oct 30 20:23:18.220814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.220816    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69\" (UniqueName: \"kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69\") pod \"143d16d4-48d0-4257-9d59-3f475da2c68d\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") "
Oct 30 20:23:18.221081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.220921    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca\") pod \"143d16d4-48d0-4257-9d59-3f475da2c68d\" (UID: \"143d16d4-48d0-4257-9d59-3f475da2c68d\") "
Oct 30 20:23:18.221228 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.221197    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca" pod "143d16d4-48d0-4257-9d59-3f475da2c68d" (UID: "143d16d4-48d0-4257-9d59-3f475da2c68d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:18.221372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.221295    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:18.223075 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.223047    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca" pod "143d16d4-48d0-4257-9d59-3f475da2c68d" (UID: "143d16d4-48d0-4257-9d59-3f475da2c68d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:18.223167 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.223105    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69" pod "143d16d4-48d0-4257-9d59-3f475da2c68d" (UID: "143d16d4-48d0-4257-9d59-3f475da2c68d"). InnerVolumeSpecName "kube-api-access-5tj69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:18.322549 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.322514    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tj69\" (UniqueName: \"kubernetes.io/projected/143d16d4-48d0-4257-9d59-3f475da2c68d-kube-api-access-5tj69\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:18.322549 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.322544    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143d16d4-48d0-4257-9d59-3f475da2c68d-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:18.899743 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.899711    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"
Oct 30 20:23:18.899941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.899744    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp" event={"ID":"143d16d4-48d0-4257-9d59-3f475da2c68d","Type":"ContainerDied","Data":"b8bee2a712b5056b99e867adfb11e8db058fed2ed12205ce4fa594ebc31ed88b"}
Oct 30 20:23:18.899941 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:18.899834    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8bee2a712b5056b99e867adfb11e8db058fed2ed12205ce4fa594ebc31ed88b"
Oct 30 20:23:19.584599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:19.584566    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"]
Oct 30 20:23:20.725654 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:20.725612    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"]
Oct 30 20:23:20.726113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:20.725921    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy" containerID="cri-o://24dae2a9567687f48d32a085cc87ecdb66608a899d5d562d25510ff489767602" gracePeriod=30
Oct 30 20:23:20.760846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:20.760800    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"]
Oct 30 20:23:20.761053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:20.761029    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy" containerID="cri-o://ade50926f260e0f473fe812db896be58b06f983f8ceafa7c6c3da527eeb784ff" gracePeriod=30
Oct 30 20:23:20.768547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:20.768515    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"]
Oct 30 20:23:20.768716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:20.768697    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy" containerID="cri-o://1a65ad88ee127938c7b4bb2804a4232989e95d6ef3eba47bbf2a27ae07616f8f" gracePeriod=30
Oct 30 20:23:21.281411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:21.281371    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:23:22.931211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:22.931187    4917 generic.go:356] "Generic (PLEG): container finished" podID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerID="35c8d37fb7ce36612f5f84189a3ef31b61d864de14cf6d9592f6bacf5bfd8872" exitCode=0
Oct 30 20:23:22.931561 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:22.931258    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" event={"ID":"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27","Type":"ContainerDied","Data":"35c8d37fb7ce36612f5f84189a3ef31b61d864de14cf6d9592f6bacf5bfd8872"}
Oct 30 20:23:23.080214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.080153    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:23:23.180838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.180808    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.180861    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.180888    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.180970    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181009 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.180998    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181018    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181096    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.181180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181106    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181202    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181273    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w\") pod \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\" (UID: \"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27\") "
Oct 30 20:23:23.181328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181300    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.181328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181317    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.181590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181565    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:23.181660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181633    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.181952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181931    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.182049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181958    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.182049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181973    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.182049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.181988    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.182049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.182002    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.183190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.183159    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.183434 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.183412    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:23.183699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.183671    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:23.183699 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.183688    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w" pod "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" (UID: "cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27"). InnerVolumeSpecName "kube-api-access-q8c7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:23.283163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.283129    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.283163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.283163    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8c7w\" (UniqueName: \"kubernetes.io/projected/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-kube-api-access-q8c7w\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.283163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.283181    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.283483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.283193    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.940405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.940372    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"
Oct 30 20:23:23.940919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.940411    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc" event={"ID":"cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27","Type":"ContainerDied","Data":"3c742e5012976481a58222855e345534ab8640f59d5a6b50d0136164afda61be"}
Oct 30 20:23:23.940919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:23.940484    4917 scope.go:122] "RemoveContainer" containerID="35c8d37fb7ce36612f5f84189a3ef31b61d864de14cf6d9592f6bacf5bfd8872"
Oct 30 20:23:24.159709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:24.159674    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"]
Oct 30 20:23:24.210297 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:24.210220    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-kbxpc"]
Oct 30 20:23:25.965675 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:25.965643    4917 generic.go:356] "Generic (PLEG): container finished" podID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerID="1a65ad88ee127938c7b4bb2804a4232989e95d6ef3eba47bbf2a27ae07616f8f" exitCode=0
Oct 30 20:23:25.966358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:25.965808    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" event={"ID":"fbeb3d71-c8f2-408f-b54a-773d051fdb23","Type":"ContainerDied","Data":"1a65ad88ee127938c7b4bb2804a4232989e95d6ef3eba47bbf2a27ae07616f8f"}
Oct 30 20:23:25.968551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:25.968529    4917 generic.go:356] "Generic (PLEG): container finished" podID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerID="24dae2a9567687f48d32a085cc87ecdb66608a899d5d562d25510ff489767602" exitCode=0
Oct 30 20:23:25.968661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:25.968575    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" event={"ID":"23902d2f-c673-49a6-acb5-514f55fa77cb","Type":"ContainerDied","Data":"24dae2a9567687f48d32a085cc87ecdb66608a899d5d562d25510ff489767602"}
Oct 30 20:23:25.971762 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:25.971740    4917 generic.go:356] "Generic (PLEG): container finished" podID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerID="ade50926f260e0f473fe812db896be58b06f983f8ceafa7c6c3da527eeb784ff" exitCode=0
Oct 30 20:23:25.971923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:25.971904    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" event={"ID":"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97","Type":"ContainerDied","Data":"ade50926f260e0f473fe812db896be58b06f983f8ceafa7c6c3da527eeb784ff"}
Oct 30 20:23:26.011400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.011313    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" path="/var/lib/kubelet/pods/cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27/volumes"
Oct 30 20:23:26.058128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.058099    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:23:26.082549 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.082522    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:23:26.104408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.104386    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:23:26.116881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.116854    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.116997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.116891    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.116997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.116910    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.116997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.116942    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.116997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.116961    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.116997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.116990    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117016    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117044    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117088    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117143    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117190    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117218    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.117272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117253    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117302    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117330    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117378    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117429    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117459    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117507    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117539    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117569    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.117643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117629    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117663    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117715    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117154    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117238    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117397    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117785    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy\") pod \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\" (UID: \"fbeb3d71-c8f2-408f-b54a-773d051fdb23\") "
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117981    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.118000    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket\") pod \"23902d2f-c673-49a6-acb5-514f55fa77cb\" (UID: \"23902d2f-c673-49a6-acb5-514f55fa77cb\") "
Oct 30 20:23:26.118179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.118047    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k\") pod \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\" (UID: \"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97\") "
Oct 30 20:23:26.118658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.118640    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.118718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.118666    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.118718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.118683    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.118718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.118698    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.119852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.119826    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "kube-api-access-26fr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.119852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.119830    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.120041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.119853    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.120041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117484    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117634    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117714    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.120041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.117417    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120114    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120132    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120125    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120154    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "kube-api-access-jqq8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.120315 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120292    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120340    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120342    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120552    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.120907 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120880    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.121024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.120994    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.121148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.121019    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy" pod "23902d2f-c673-49a6-acb5-514f55fa77cb" (UID: "23902d2f-c673-49a6-acb5-514f55fa77cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.121431 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.121407    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "kube-api-access-gx78k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.122097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.122078    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.122389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.122375    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token" pod "fbeb3d71-c8f2-408f-b54a-773d051fdb23" (UID: "fbeb3d71-c8f2-408f-b54a-773d051fdb23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.122454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.122435    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.122568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.122557    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token" pod "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" (UID: "59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.220042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220008    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx78k\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-kube-api-access-gx78k\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220042 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220040    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220050    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fr2\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-kube-api-access-26fr2\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220059    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220069    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220078    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqq8n\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-kube-api-access-jqq8n\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220087    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220097    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220109    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23902d2f-c673-49a6-acb5-514f55fa77cb-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220118    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23902d2f-c673-49a6-acb5-514f55fa77cb-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220126    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220134    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220144    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220153    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220162    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220171    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220185    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220195    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220204    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220212    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220220    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220228    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbeb3d71-c8f2-408f-b54a-773d051fdb23-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.220253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.220236    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23902d2f-c673-49a6-acb5-514f55fa77cb-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.685467 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.685429    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"]
Oct 30 20:23:26.800380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.800345    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"]
Oct 30 20:23:26.979640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.979558    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"
Oct 30 20:23:26.979640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.979596    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh" event={"ID":"59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97","Type":"ContainerDied","Data":"29cc96f42b7f5f5976a3b3648e36ec27acccbbbf321c8f3ce7b917d621b2026e"}
Oct 30 20:23:26.980217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.979655    4917 scope.go:122] "RemoveContainer" containerID="ade50926f260e0f473fe812db896be58b06f983f8ceafa7c6c3da527eeb784ff"
Oct 30 20:23:26.981160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.981083    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"
Oct 30 20:23:26.981160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.981129    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn" event={"ID":"fbeb3d71-c8f2-408f-b54a-773d051fdb23","Type":"ContainerDied","Data":"d367cf98629b48da60e08ed676ab98128ebed5d0048a71ed7ff9396b37b9de2b"}
Oct 30 20:23:26.982911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.982889    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"
Oct 30 20:23:26.983025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.982883    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn" event={"ID":"23902d2f-c673-49a6-acb5-514f55fa77cb","Type":"ContainerDied","Data":"846b0ca142012b8a13b01513ce81da055d572dcf6f2456282bc8da2cadcd692c"}
Oct 30 20:23:26.996760 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:26.996736    4917 scope.go:122] "RemoveContainer" containerID="1a65ad88ee127938c7b4bb2804a4232989e95d6ef3eba47bbf2a27ae07616f8f"
Oct 30 20:23:27.020948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.020924    4917 scope.go:122] "RemoveContainer" containerID="24dae2a9567687f48d32a085cc87ecdb66608a899d5d562d25510ff489767602"
Oct 30 20:23:27.155191 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.155158    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"]
Oct 30 20:23:27.159480 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.159454    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nczwn"]
Oct 30 20:23:27.273615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.273493    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"]
Oct 30 20:23:27.278182 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.278153    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nhprh"]
Oct 30 20:23:27.390324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.390293    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"]
Oct 30 20:23:27.395841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:27.395814    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-jsgpn"]
Oct 30 20:23:28.001713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.001677    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" path="/var/lib/kubelet/pods/23902d2f-c673-49a6-acb5-514f55fa77cb/volumes"
Oct 30 20:23:28.002243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.002228    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" path="/var/lib/kubelet/pods/59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97/volumes"
Oct 30 20:23:28.002662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.002649    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" path="/var/lib/kubelet/pods/fbeb3d71-c8f2-408f-b54a-773d051fdb23/volumes"
Oct 30 20:23:28.051867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.051834    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"]
Oct 30 20:23:28.052323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052309    4917 cpu_manager.go:414] "containerMap: removing container" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052324    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052337    4917 cpu_manager.go:414] "containerMap: removing container" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052342    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052354    4917 cpu_manager.go:414] "containerMap: removing container" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052360    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052367    4917 cpu_manager.go:414] "containerMap: removing container" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy"
Oct 30 20:23:28.052372 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052371    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052389    4917 cpu_manager.go:414] "containerMap: removing container" podUID="143d16d4-48d0-4257-9d59-3f475da2c68d" containerName="revoke-keys"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052394    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143d16d4-48d0-4257-9d59-3f475da2c68d" containerName="revoke-keys"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052476    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeb3d71-c8f2-408f-b54a-773d051fdb23" containerName="istio-proxy"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052487    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="23902d2f-c673-49a6-acb5-514f55fa77cb" containerName="istio-proxy"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052494    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ba07f1-cec9-44f4-8e4d-bbf41d5c9f97" containerName="istio-proxy"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052501    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda9fa23-1d23-4ea8-b94b-ce4c58ce7b27" containerName="istio-proxy"
Oct 30 20:23:28.052581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.052509    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="143d16d4-48d0-4257-9d59-3f475da2c68d" containerName="revoke-keys"
Oct 30 20:23:28.057528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.057511    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.111986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.111564    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:28.111986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.111644    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:28.111986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.111738    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:23:28.111986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.111808    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:28.116225 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.116202    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"]
Oct 30 20:23:28.143972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.143411    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.143972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.143470    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwv4g\" (UniqueName: \"kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.143972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.143556    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.244886 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.244852    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.245076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.244994    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.245076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.245021    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwv4g\" (UniqueName: \"kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.245563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.245542    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.247516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.247489    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.309597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.309519    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwv4g\" (UniqueName: \"kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g\") pod \"maas-api-revoke-sub-ac56a5bd-77whx\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.371232 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.371198    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:28.769534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:28.769508    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"]
Oct 30 20:23:28.770714 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:23:28.770690    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b64805_fa8c_4882_b3de_882cc26ad733.slice/crio-5d3464c72c79c2e935afb7d0ceaf68d9a68ac7a9c65e747cb4056c5ef4d30ffc WatchSource:0}: Error finding container 5d3464c72c79c2e935afb7d0ceaf68d9a68ac7a9c65e747cb4056c5ef4d30ffc: Status 404 returned error can't find the container with id 5d3464c72c79c2e935afb7d0ceaf68d9a68ac7a9c65e747cb4056c5ef4d30ffc
Oct 30 20:23:29.003077 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:29.003047    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx" event={"ID":"18b64805-fa8c-4882-b3de-882cc26ad733","Type":"ContainerStarted","Data":"5d3464c72c79c2e935afb7d0ceaf68d9a68ac7a9c65e747cb4056c5ef4d30ffc"}
Oct 30 20:23:30.015741 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:30.015658    4917 generic.go:356] "Generic (PLEG): container finished" podID="18b64805-fa8c-4882-b3de-882cc26ad733" containerID="b477bd2ee9489faf8b9c864c7e65532ed057d20d06d4951156843bef3f6b7c20" exitCode=0
Oct 30 20:23:30.015741 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:30.015714    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx" event={"ID":"18b64805-fa8c-4882-b3de-882cc26ad733","Type":"ContainerDied","Data":"b477bd2ee9489faf8b9c864c7e65532ed057d20d06d4951156843bef3f6b7c20"}
Oct 30 20:23:31.198021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.197993    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:31.278729 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.278646    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca\") pod \"18b64805-fa8c-4882-b3de-882cc26ad733\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") "
Oct 30 20:23:31.278905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.278853    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g\" (UniqueName: \"kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g\") pod \"18b64805-fa8c-4882-b3de-882cc26ad733\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") "
Oct 30 20:23:31.278968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.278908    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca\") pod \"18b64805-fa8c-4882-b3de-882cc26ad733\" (UID: \"18b64805-fa8c-4882-b3de-882cc26ad733\") "
Oct 30 20:23:31.279308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.279281    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca" pod "18b64805-fa8c-4882-b3de-882cc26ad733" (UID: "18b64805-fa8c-4882-b3de-882cc26ad733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:31.280847 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.280823    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca" pod "18b64805-fa8c-4882-b3de-882cc26ad733" (UID: "18b64805-fa8c-4882-b3de-882cc26ad733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:31.280978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.280957    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g" pod "18b64805-fa8c-4882-b3de-882cc26ad733" (UID: "18b64805-fa8c-4882-b3de-882cc26ad733"). InnerVolumeSpecName "kube-api-access-dwv4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:31.380580 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.380541    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwv4g\" (UniqueName: \"kubernetes.io/projected/18b64805-fa8c-4882-b3de-882cc26ad733-kube-api-access-dwv4g\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:31.380580 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.380572    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:31.380796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.380587    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b64805-fa8c-4882-b3de-882cc26ad733-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:31.686793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.686728    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerName="payload-processing" containerID="cri-o://f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb" gracePeriod=25
Oct 30 20:23:31.801229 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:31.801185    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" containerName="payload-pre-processing" containerID="cri-o://036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb" gracePeriod=25
Oct 30 20:23:32.031864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:32.031831    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx" event={"ID":"18b64805-fa8c-4882-b3de-882cc26ad733","Type":"ContainerDied","Data":"5d3464c72c79c2e935afb7d0ceaf68d9a68ac7a9c65e747cb4056c5ef4d30ffc"}
Oct 30 20:23:32.032030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:32.031880    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d3464c72c79c2e935afb7d0ceaf68d9a68ac7a9c65e747cb4056c5ef4d30ffc"
Oct 30 20:23:32.032030 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:32.031883    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"
Oct 30 20:23:34.605152 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.605110    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" containerID="cri-o://08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6" gracePeriod=45
Oct 30 20:23:34.683436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.683402    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 20:23:34.855727 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.855658    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:23:34.920515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920486    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache\") pod \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") "
Oct 30 20:23:34.920697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920554    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs\") pod \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") "
Oct 30 20:23:34.920791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920704    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home\") pod \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") "
Oct 30 20:23:34.920791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920736    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l\" (UniqueName: \"kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l\") pod \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") "
Oct 30 20:23:34.920791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920738    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache" pod "edbeb7ac-aabc-4335-803a-def6b3eaa5bd" (UID: "edbeb7ac-aabc-4335-803a-def6b3eaa5bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.920791 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920788    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir\") pod \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") "
Oct 30 20:23:34.921010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920848    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm\") pod \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\" (UID: \"edbeb7ac-aabc-4335-803a-def6b3eaa5bd\") "
Oct 30 20:23:34.921010 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.920955    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home" pod "edbeb7ac-aabc-4335-803a-def6b3eaa5bd" (UID: "edbeb7ac-aabc-4335-803a-def6b3eaa5bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.921101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.921040    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir" pod "edbeb7ac-aabc-4335-803a-def6b3eaa5bd" (UID: "edbeb7ac-aabc-4335-803a-def6b3eaa5bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.921258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.921233    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.921258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.921257    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.921462 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.921267    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.923063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.923037    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm" pod "edbeb7ac-aabc-4335-803a-def6b3eaa5bd" (UID: "edbeb7ac-aabc-4335-803a-def6b3eaa5bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.923148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.923073    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs" pod "edbeb7ac-aabc-4335-803a-def6b3eaa5bd" (UID: "edbeb7ac-aabc-4335-803a-def6b3eaa5bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:34.923185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:34.923159    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l" pod "edbeb7ac-aabc-4335-803a-def6b3eaa5bd" (UID: "edbeb7ac-aabc-4335-803a-def6b3eaa5bd"). InnerVolumeSpecName "kube-api-access-cx79l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:35.022389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.022357    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.022389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.022388    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx79l\" (UniqueName: \"kubernetes.io/projected/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-kube-api-access-cx79l\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.022590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.022401    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edbeb7ac-aabc-4335-803a-def6b3eaa5bd-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.085116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.085070    4917 generic.go:356] "Generic (PLEG): container finished" podID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerID="08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6" exitCode=0
Oct 30 20:23:35.085267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.085179    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" event={"ID":"edbeb7ac-aabc-4335-803a-def6b3eaa5bd","Type":"ContainerDied","Data":"08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6"}
Oct 30 20:23:35.085267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.085220    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd" event={"ID":"edbeb7ac-aabc-4335-803a-def6b3eaa5bd","Type":"ContainerDied","Data":"32d268f04839fd418bdbaceb124550ac8646bcd485e7e5efc123626e9526a3f2"}
Oct 30 20:23:35.085267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.085244    4917 scope.go:122] "RemoveContainer" containerID="08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6"
Oct 30 20:23:35.085433 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.085418    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"
Oct 30 20:23:35.099151 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.099126    4917 scope.go:122] "RemoveContainer" containerID="08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6"
Oct 30 20:23:35.099446 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:23:35.099425    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6\": container with ID starting with 08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6 not found: ID does not exist" containerID="08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6"
Oct 30 20:23:35.099522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.099460    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6"} err="failed to get container status \"08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6\": rpc error: code = NotFound desc = could not find container \"08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6\": container with ID starting with 08c04bd1252a7ad32ceda08edc2e74ed44b28d6b0cb9a44d2d28a35b04751ae6 not found: ID does not exist"
Oct 30 20:23:35.256589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.256554    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"]
Oct 30 20:23:35.262725 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:35.262695    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-w4-bebb42-kserve-7995c4cff8-qp5nd"]
Oct 30 20:23:36.002627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:36.002597    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" path="/var/lib/kubelet/pods/edbeb7ac-aabc-4335-803a-def6b3eaa5bd/volumes"
Oct 30 20:23:39.058672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.058641    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"]
Oct 30 20:23:39.059202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.059186    4917 cpu_manager.go:414] "containerMap: removing container" podUID="18b64805-fa8c-4882-b3de-882cc26ad733" containerName="revoke-keys"
Oct 30 20:23:39.059257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.059205    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b64805-fa8c-4882-b3de-882cc26ad733" containerName="revoke-keys"
Oct 30 20:23:39.059257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.059234    4917 cpu_manager.go:414] "containerMap: removing container" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main"
Oct 30 20:23:39.059257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.059240    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main"
Oct 30 20:23:39.059373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.059344    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b64805-fa8c-4882-b3de-882cc26ad733" containerName="revoke-keys"
Oct 30 20:23:39.059373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.059358    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbeb7ac-aabc-4335-803a-def6b3eaa5bd" containerName="main"
Oct 30 20:23:39.064113 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.064093    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.119499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.119472    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:39.169520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.169485    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:23:39.169520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.169504    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:23:39.169520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.169512    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:23:39.169766 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.169673    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.169851 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.169829    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb628\" (UniqueName: \"kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.170003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.169981    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.172614 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.172592    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"]
Oct 30 20:23:39.270959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.270924    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb628\" (UniqueName: \"kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.271145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.271049    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.271243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.271143    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.271832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.271810    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.273531 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.273514    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.339642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.339573    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb628\" (UniqueName: \"kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628\") pod \"maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.377544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.377519    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:39.527242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.527200    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:39.527435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.527259    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:39.679833 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:23:39.679798    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:23:39.754291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:39.754265    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"]
Oct 30 20:23:39.755305 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:23:39.755275    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416fd293_cf90_4b96_842d_a56d279807bc.slice/crio-ac51bbc4d301ac5d9a8da544331636106c3ac1f4cf0dd025df536de5c5c3029b WatchSource:0}: Error finding container ac51bbc4d301ac5d9a8da544331636106c3ac1f4cf0dd025df536de5c5c3029b: Status 404 returned error can't find the container with id ac51bbc4d301ac5d9a8da544331636106c3ac1f4cf0dd025df536de5c5c3029b
Oct 30 20:23:40.121515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:40.121478    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk" event={"ID":"416fd293-cf90-4b96-842d-a56d279807bc","Type":"ContainerStarted","Data":"f7311183398d8a860ffc146aeae44ba5ade7102fcca9c9f639aa47ebedc3d77c"}
Oct 30 20:23:40.122059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:40.121540    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk" event={"ID":"416fd293-cf90-4b96-842d-a56d279807bc","Type":"ContainerStarted","Data":"ac51bbc4d301ac5d9a8da544331636106c3ac1f4cf0dd025df536de5c5c3029b"}
Oct 30 20:23:41.130457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:41.130423    4917 generic.go:356] "Generic (PLEG): container finished" podID="416fd293-cf90-4b96-842d-a56d279807bc" containerID="f7311183398d8a860ffc146aeae44ba5ade7102fcca9c9f639aa47ebedc3d77c" exitCode=0
Oct 30 20:23:41.131004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:41.130514    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk" event={"ID":"416fd293-cf90-4b96-842d-a56d279807bc","Type":"ContainerDied","Data":"f7311183398d8a860ffc146aeae44ba5ade7102fcca9c9f639aa47ebedc3d77c"}
Oct 30 20:23:42.314317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.314286    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:42.411468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.411375    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca\") pod \"416fd293-cf90-4b96-842d-a56d279807bc\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") "
Oct 30 20:23:42.411468 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.411464    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca\") pod \"416fd293-cf90-4b96-842d-a56d279807bc\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") "
Oct 30 20:23:42.411707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.411595    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628\" (UniqueName: \"kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628\") pod \"416fd293-cf90-4b96-842d-a56d279807bc\" (UID: \"416fd293-cf90-4b96-842d-a56d279807bc\") "
Oct 30 20:23:42.411810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.411716    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca" pod "416fd293-cf90-4b96-842d-a56d279807bc" (UID: "416fd293-cf90-4b96-842d-a56d279807bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:42.412126 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.412102    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:42.413825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.413763    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca" pod "416fd293-cf90-4b96-842d-a56d279807bc" (UID: "416fd293-cf90-4b96-842d-a56d279807bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:42.413938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.413919    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628" pod "416fd293-cf90-4b96-842d-a56d279807bc" (UID: "416fd293-cf90-4b96-842d-a56d279807bc"). InnerVolumeSpecName "kube-api-access-xb628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:42.513155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.513117    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416fd293-cf90-4b96-842d-a56d279807bc-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:42.513155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:42.513149    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb628\" (UniqueName: \"kubernetes.io/projected/416fd293-cf90-4b96-842d-a56d279807bc-kube-api-access-xb628\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:43.145581 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:43.145548    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"
Oct 30 20:23:43.145733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:43.145550    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk" event={"ID":"416fd293-cf90-4b96-842d-a56d279807bc","Type":"ContainerDied","Data":"ac51bbc4d301ac5d9a8da544331636106c3ac1f4cf0dd025df536de5c5c3029b"}
Oct 30 20:23:43.145733 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:43.145696    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac51bbc4d301ac5d9a8da544331636106c3ac1f4cf0dd025df536de5c5c3029b"
Oct 30 20:23:43.681861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:43.681828    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"]
Oct 30 20:23:45.552746 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:45.552717    4917 scope.go:122] "RemoveContainer" containerID="537aebc6900c1658c5eaecfe10d9cb78b336c1fd4f4efbd3c1990aace5340a2f"
Oct 30 20:23:45.562823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:45.562764    4917 scope.go:122] "RemoveContainer" containerID="419fca8092562d58d5db9b8384a19ba1a8747d09845aeaa7eab7e9a54ae33169"
Oct 30 20:23:45.572131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:45.572113    4917 scope.go:122] "RemoveContainer" containerID="1af0e18194425b3b2f3291814220d76de20247b96492459543302e3d28548ddc"
Oct 30 20:23:45.580613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:45.580589    4917 scope.go:122] "RemoveContainer" containerID="2df74d4cbe7bbeef1a00697ef8f94b4cbd93d73b349be494d06bc96f053b5b73"
Oct 30 20:23:47.399307 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:47.399266    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:23:49.058055 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:49.058013    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"]
Oct 30 20:23:49.104353 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:49.104296    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713b5ac-c6txr"]
Oct 30 20:23:49.527189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:49.527157    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:49.527367 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:49.527152    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:23:50.004535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:50.004500    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea5515d-53ac-47a8-883b-7d22385252f7" path="/var/lib/kubelet/pods/fea5515d-53ac-47a8-883b-7d22385252f7/volumes"
Oct 30 20:23:51.966978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:51.966911    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 20:23:52.070248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.070223    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 20:23:52.126264 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.126235    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume\") pod \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") "
Oct 30 20:23:52.126441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.126339    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9\" (UniqueName: \"kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9\") pod \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\" (UID: \"1d5bbf08-ce59-4b13-af66-297f73dab2d5\") "
Oct 30 20:23:52.126621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.126597    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume" pod "1d5bbf08-ce59-4b13-af66-297f73dab2d5" (UID: "1d5bbf08-ce59-4b13-af66-297f73dab2d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:52.128463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.128442    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9" pod "1d5bbf08-ce59-4b13-af66-297f73dab2d5" (UID: "1d5bbf08-ce59-4b13-af66-297f73dab2d5"). InnerVolumeSpecName "kube-api-access-8tsq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:52.206470 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.206439    4917 generic.go:356] "Generic (PLEG): container finished" podID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerID="f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb" exitCode=0
Oct 30 20:23:52.206633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.206479    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"
Oct 30 20:23:52.206633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.206524    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" event={"ID":"1d5bbf08-ce59-4b13-af66-297f73dab2d5","Type":"ContainerDied","Data":"f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb"}
Oct 30 20:23:52.206633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.206576    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr" event={"ID":"1d5bbf08-ce59-4b13-af66-297f73dab2d5","Type":"ContainerDied","Data":"187191dade308425c69cba1c646ce99a98398a07efb86c3a03237e10b5fdbaaa"}
Oct 30 20:23:52.206633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.206602    4917 scope.go:122] "RemoveContainer" containerID="f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb"
Oct 30 20:23:52.208636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.208587    4917 generic.go:356] "Generic (PLEG): container finished" podID="2a213d8a-0c63-4487-a624-1ae629078a20" containerID="036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb" exitCode=0
Oct 30 20:23:52.208923 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.208898    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" event={"ID":"2a213d8a-0c63-4487-a624-1ae629078a20","Type":"ContainerDied","Data":"036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb"}
Oct 30 20:23:52.209050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.208948    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb" event={"ID":"2a213d8a-0c63-4487-a624-1ae629078a20","Type":"ContainerDied","Data":"d58406b60498acd0bfffabc2f3e814a7cbca7101ed8993fac5230ace0c0e46ac"}
Oct 30 20:23:52.210180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.210158    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"
Oct 30 20:23:52.223342 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.223319    4917 scope.go:122] "RemoveContainer" containerID="f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb"
Oct 30 20:23:52.223651 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:23:52.223630    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb\": container with ID starting with f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb not found: ID does not exist" containerID="f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb"
Oct 30 20:23:52.223723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.223663    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb"} err="failed to get container status \"f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb\": rpc error: code = NotFound desc = could not find container \"f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb\": container with ID starting with f66efff42094a50c75ebef5c1b0c8dc19366ff4864ec5b4b322d474932c280eb not found: ID does not exist"
Oct 30 20:23:52.223723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.223688    4917 scope.go:122] "RemoveContainer" containerID="036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb"
Oct 30 20:23:52.227200 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.227170    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8\" (UniqueName: \"kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8\") pod \"2a213d8a-0c63-4487-a624-1ae629078a20\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") "
Oct 30 20:23:52.227313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.227266    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume\") pod \"2a213d8a-0c63-4487-a624-1ae629078a20\" (UID: \"2a213d8a-0c63-4487-a624-1ae629078a20\") "
Oct 30 20:23:52.227626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.227604    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume" pod "2a213d8a-0c63-4487-a624-1ae629078a20" (UID: "2a213d8a-0c63-4487-a624-1ae629078a20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:52.227764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.227744    4917 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a213d8a-0c63-4487-a624-1ae629078a20-plugins-config-volume\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:52.227864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.227810    4917 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5bbf08-ce59-4b13-af66-297f73dab2d5-plugins-config-volume\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:52.227864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.227829    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsq9\" (UniqueName: \"kubernetes.io/projected/1d5bbf08-ce59-4b13-af66-297f73dab2d5-kube-api-access-8tsq9\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:52.229269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.229238    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8" pod "2a213d8a-0c63-4487-a624-1ae629078a20" (UID: "2a213d8a-0c63-4487-a624-1ae629078a20"). InnerVolumeSpecName "kube-api-access-bs6b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:52.234845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.234829    4917 scope.go:122] "RemoveContainer" containerID="036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb"
Oct 30 20:23:52.235124 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:23:52.235092    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb\": container with ID starting with 036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb not found: ID does not exist" containerID="036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb"
Oct 30 20:23:52.235218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.235133    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb"} err="failed to get container status \"036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb\": rpc error: code = NotFound desc = could not find container \"036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb\": container with ID starting with 036582f443e08a63762469cfe6279caae369276db3bdb6fd85810f80ab847edb not found: ID does not exist"
Oct 30 20:23:52.336308 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.336250    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs6b8\" (UniqueName: \"kubernetes.io/projected/2a213d8a-0c63-4487-a624-1ae629078a20-kube-api-access-bs6b8\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:52.430995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.430955    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"]
Oct 30 20:23:52.435025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.434993    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebb42-7d5f8f7b9b-5zmdr"]
Oct 30 20:23:52.652002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.651276    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"]
Oct 30 20:23:52.655142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:52.655113    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebb42-6596745dcf-zk6mb"]
Oct 30 20:23:54.002757 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:54.002722    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" path="/var/lib/kubelet/pods/1d5bbf08-ce59-4b13-af66-297f73dab2d5/volumes"
Oct 30 20:23:54.003234 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:54.003218    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" path="/var/lib/kubelet/pods/2a213d8a-0c63-4487-a624-1ae629078a20/volumes"
Oct 30 20:23:58.702846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:58.702790    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" containerID="cri-o://cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc" gracePeriod=45
Oct 30 20:23:58.968939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:58.968876    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:23:59.021795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.021735    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache\") pod \"05e3bdda-e405-4a41-97b0-9b67005898e8\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") "
Oct 30 20:23:59.021956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.021838    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs\") pod \"05e3bdda-e405-4a41-97b0-9b67005898e8\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") "
Oct 30 20:23:59.021956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.021870    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home\") pod \"05e3bdda-e405-4a41-97b0-9b67005898e8\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") "
Oct 30 20:23:59.021956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.021952    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm\") pod \"05e3bdda-e405-4a41-97b0-9b67005898e8\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") "
Oct 30 20:23:59.022127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.021995    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l\" (UniqueName: \"kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l\") pod \"05e3bdda-e405-4a41-97b0-9b67005898e8\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") "
Oct 30 20:23:59.022127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022028    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache" pod "05e3bdda-e405-4a41-97b0-9b67005898e8" (UID: "05e3bdda-e405-4a41-97b0-9b67005898e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.022127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022052    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir\") pod \"05e3bdda-e405-4a41-97b0-9b67005898e8\" (UID: \"05e3bdda-e405-4a41-97b0-9b67005898e8\") "
Oct 30 20:23:59.022287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022170    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home" pod "05e3bdda-e405-4a41-97b0-9b67005898e8" (UID: "05e3bdda-e405-4a41-97b0-9b67005898e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.022379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022353    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir" pod "05e3bdda-e405-4a41-97b0-9b67005898e8" (UID: "05e3bdda-e405-4a41-97b0-9b67005898e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.022547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022526    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.022617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022551    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.022617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.022581    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.024171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.024148    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm" pod "05e3bdda-e405-4a41-97b0-9b67005898e8" (UID: "05e3bdda-e405-4a41-97b0-9b67005898e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.024436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.024413    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs" pod "05e3bdda-e405-4a41-97b0-9b67005898e8" (UID: "05e3bdda-e405-4a41-97b0-9b67005898e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:59.024514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.024417    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l" pod "05e3bdda-e405-4a41-97b0-9b67005898e8" (UID: "05e3bdda-e405-4a41-97b0-9b67005898e8"). InnerVolumeSpecName "kube-api-access-m4b2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:59.124186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.124151    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e3bdda-e405-4a41-97b0-9b67005898e8-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.124186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.124182    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4b2l\" (UniqueName: \"kubernetes.io/projected/05e3bdda-e405-4a41-97b0-9b67005898e8-kube-api-access-m4b2l\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.124186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.124192    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e3bdda-e405-4a41-97b0-9b67005898e8-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.264436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.264343    4917 generic.go:356] "Generic (PLEG): container finished" podID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerID="cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc" exitCode=0
Oct 30 20:23:59.264436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.264396    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"
Oct 30 20:23:59.264650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.264441    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" event={"ID":"05e3bdda-e405-4a41-97b0-9b67005898e8","Type":"ContainerDied","Data":"cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc"}
Oct 30 20:23:59.264650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.264494    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" event={"ID":"05e3bdda-e405-4a41-97b0-9b67005898e8","Type":"ContainerDied","Data":"4fff1cc50990c25b13900fdd7d6e2aed17955d3ec4f525209ccf658b427a0972"}
Oct 30 20:23:59.264650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.264518    4917 scope.go:122] "RemoveContainer" containerID="cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc"
Oct 30 20:23:59.277985 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.277960    4917 scope.go:122] "RemoveContainer" containerID="cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc"
Oct 30 20:23:59.278294 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:23:59.278271    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc\": container with ID starting with cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc not found: ID does not exist" containerID="cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc"
Oct 30 20:23:59.278394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.278305    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc"} err="failed to get container status \"cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc\": rpc error: code = NotFound desc = could not find container \"cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc\": container with ID starting with cac3bec3c2793bbb4b2ddeee581b634beb102c8ae23be2523c2cb640d93a65dc not found: ID does not exist"
Oct 30 20:23:59.434338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.434303    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"]
Oct 30 20:23:59.453021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.452990    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r"]
Oct 30 20:23:59.960396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:23:59.960351    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/premium-w6-f75272-kserve-54749777dc-tsd5r" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:24:00.006606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:00.006576    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" path="/var/lib/kubelet/pods/05e3bdda-e405-4a41-97b0-9b67005898e8/volumes"
Oct 30 20:24:04.625383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.625344    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"]
Oct 30 20:24:04.627022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.626993    4917 cpu_manager.go:414] "containerMap: removing container" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main"
Oct 30 20:24:04.627177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627163    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main"
Oct 30 20:24:04.627304 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627293    4917 cpu_manager.go:414] "containerMap: removing container" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerName="payload-processing"
Oct 30 20:24:04.627380 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627370    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerName="payload-processing"
Oct 30 20:24:04.627461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627452    4917 cpu_manager.go:414] "containerMap: removing container" podUID="416fd293-cf90-4b96-842d-a56d279807bc" containerName="revoke-keys"
Oct 30 20:24:04.627535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627526    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416fd293-cf90-4b96-842d-a56d279807bc" containerName="revoke-keys"
Oct 30 20:24:04.627613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627604    4917 cpu_manager.go:414] "containerMap: removing container" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" containerName="payload-pre-processing"
Oct 30 20:24:04.627683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.627673    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" containerName="payload-pre-processing"
Oct 30 20:24:04.628098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.628078    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e3bdda-e405-4a41-97b0-9b67005898e8" containerName="main"
Oct 30 20:24:04.628223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.628213    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5bbf08-ce59-4b13-af66-297f73dab2d5" containerName="payload-processing"
Oct 30 20:24:04.628340 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.628329    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="416fd293-cf90-4b96-842d-a56d279807bc" containerName="revoke-keys"
Oct 30 20:24:04.628419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.628410    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a213d8a-0c63-4487-a624-1ae629078a20" containerName="payload-pre-processing"
Oct 30 20:24:04.634585 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.634562    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.686388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.686357    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"]
Oct 30 20:24:04.688097 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.688077    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.688184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.688127    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw88t\" (UniqueName: \"kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.688244 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.688221    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.733301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.733271    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:24:04.733301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.733271    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:24:04.733301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.733274    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:04.738218 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.738196    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:24:04.788955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.788921    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw88t\" (UniqueName: \"kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.788955 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.788953    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.789140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.789066    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.789691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.789666    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.791499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.791472    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.852048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.852018    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw88t\" (UniqueName: \"kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t\") pod \"maas-api-revoke-sub-221df18b-gn27p\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.866530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.866501    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"]
Oct 30 20:24:04.870365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.870346    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:04.930116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.930086    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"]
Oct 30 20:24:04.948646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.948617    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:04.992371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.991746    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:04.993908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.993617    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2bt\" (UniqueName: \"kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:04.993908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:04.993741    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.094626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.094584    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.094828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.094654    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2bt\" (UniqueName: \"kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.094828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.094765    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.095419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.095391    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.097263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.097231    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.156939 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.156874    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2bt\" (UniqueName: \"kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt\") pod \"maas-api-revoke-sub-f8c9af9e-sqjnd\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.183864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.183834    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:05.337486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.337459    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"]
Oct 30 20:24:05.338006 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:24:05.337979    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bb20b0_c159_4221_a734_103a05b35eff.slice/crio-8986f1723a1e89bdc4700392f463e1e6c78872a5d2d8b0c48cc786783387db7f WatchSource:0}: Error finding container 8986f1723a1e89bdc4700392f463e1e6c78872a5d2d8b0c48cc786783387db7f: Status 404 returned error can't find the container with id 8986f1723a1e89bdc4700392f463e1e6c78872a5d2d8b0c48cc786783387db7f
Oct 30 20:24:05.563284 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:24:05.563255    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc2c3cc_d6e7_47e0_bd10_d038ac00f6f9.slice/crio-87abe375464549633f47730ef4e3bd407deaac21aa98a018b2717d27ce761f84 WatchSource:0}: Error finding container 87abe375464549633f47730ef4e3bd407deaac21aa98a018b2717d27ce761f84: Status 404 returned error can't find the container with id 87abe375464549633f47730ef4e3bd407deaac21aa98a018b2717d27ce761f84
Oct 30 20:24:05.563426 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:05.563406    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"]
Oct 30 20:24:06.317463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:06.317422    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd" event={"ID":"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9","Type":"ContainerStarted","Data":"c4a2a950c2a9259f2541259cf7a51b078115d6644602ac8ddddc3dc9bf3ba673"}
Oct 30 20:24:06.317961 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:06.317489    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd" event={"ID":"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9","Type":"ContainerStarted","Data":"87abe375464549633f47730ef4e3bd407deaac21aa98a018b2717d27ce761f84"}
Oct 30 20:24:06.319028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:06.319006    4917 generic.go:356] "Generic (PLEG): container finished" podID="d6bb20b0-c159-4221-a734-103a05b35eff" containerID="a4a22237ecfaf3363fe85caaf8a34abed0fd10788d93a7e3c0608f5fb89af512" exitCode=0
Oct 30 20:24:06.319134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:06.319052    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p" event={"ID":"d6bb20b0-c159-4221-a734-103a05b35eff","Type":"ContainerDied","Data":"a4a22237ecfaf3363fe85caaf8a34abed0fd10788d93a7e3c0608f5fb89af512"}
Oct 30 20:24:06.319134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:06.319081    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p" event={"ID":"d6bb20b0-c159-4221-a734-103a05b35eff","Type":"ContainerStarted","Data":"8986f1723a1e89bdc4700392f463e1e6c78872a5d2d8b0c48cc786783387db7f"}
Oct 30 20:24:06.435953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:06.435908    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd" podStartSLOduration=2.435895254 podStartE2EDuration="2.435895254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:06.434302119 +0000 UTC m=+2542.598703761" watchObservedRunningTime="2026-10-30 20:24:06.435895254 +0000 UTC m=+2542.600296895"
Oct 30 20:24:07.327904 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.327872    4917 generic.go:356] "Generic (PLEG): container finished" podID="7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" containerID="c4a2a950c2a9259f2541259cf7a51b078115d6644602ac8ddddc3dc9bf3ba673" exitCode=0
Oct 30 20:24:07.328344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.327926    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd" event={"ID":"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9","Type":"ContainerDied","Data":"c4a2a950c2a9259f2541259cf7a51b078115d6644602ac8ddddc3dc9bf3ba673"}
Oct 30 20:24:07.482358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.482332    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:07.523629 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.523595    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca\") pod \"d6bb20b0-c159-4221-a734-103a05b35eff\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") "
Oct 30 20:24:07.523793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.523735    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca\") pod \"d6bb20b0-c159-4221-a734-103a05b35eff\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") "
Oct 30 20:24:07.523893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.523876    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t\" (UniqueName: \"kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t\") pod \"d6bb20b0-c159-4221-a734-103a05b35eff\" (UID: \"d6bb20b0-c159-4221-a734-103a05b35eff\") "
Oct 30 20:24:07.524000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.523977    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca" pod "d6bb20b0-c159-4221-a734-103a05b35eff" (UID: "d6bb20b0-c159-4221-a734-103a05b35eff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:07.524247 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.524227    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:07.525832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.525747    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca" pod "d6bb20b0-c159-4221-a734-103a05b35eff" (UID: "d6bb20b0-c159-4221-a734-103a05b35eff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:07.525933 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.525901    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t" pod "d6bb20b0-c159-4221-a734-103a05b35eff" (UID: "d6bb20b0-c159-4221-a734-103a05b35eff"). InnerVolumeSpecName "kube-api-access-tw88t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:07.625749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.625649    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6bb20b0-c159-4221-a734-103a05b35eff-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:07.625749 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:07.625680    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw88t\" (UniqueName: \"kubernetes.io/projected/d6bb20b0-c159-4221-a734-103a05b35eff-kube-api-access-tw88t\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.336305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.336197    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"
Oct 30 20:24:08.336305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.336231    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p" event={"ID":"d6bb20b0-c159-4221-a734-103a05b35eff","Type":"ContainerDied","Data":"8986f1723a1e89bdc4700392f463e1e6c78872a5d2d8b0c48cc786783387db7f"}
Oct 30 20:24:08.336305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.336282    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8986f1723a1e89bdc4700392f463e1e6c78872a5d2d8b0c48cc786783387db7f"
Oct 30 20:24:08.476953 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.476925    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:08.534701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.534666    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt\" (UniqueName: \"kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt\") pod \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") "
Oct 30 20:24:08.534902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.534813    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca\") pod \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") "
Oct 30 20:24:08.534975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.534958    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca\") pod \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\" (UID: \"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9\") "
Oct 30 20:24:08.535217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.535188    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca" pod "7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" (UID: "7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:08.535466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.535441    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.537226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.537196    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca" pod "7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" (UID: "7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:08.537338 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.537282    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt" pod "7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" (UID: "7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9"). InnerVolumeSpecName "kube-api-access-xl2bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:08.637282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.637208    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.637282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:08.637246    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl2bt\" (UniqueName: \"kubernetes.io/projected/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9-kube-api-access-xl2bt\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:09.344515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:09.344481    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"
Oct 30 20:24:09.344515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:09.344501    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd" event={"ID":"7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9","Type":"ContainerDied","Data":"87abe375464549633f47730ef4e3bd407deaac21aa98a018b2717d27ce761f84"}
Oct 30 20:24:09.345011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:09.344558    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87abe375464549633f47730ef4e3bd407deaac21aa98a018b2717d27ce761f84"
Oct 30 20:24:09.680193 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:24:09.680161    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:24:39.679515 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:24:39.679434    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:24:41.138897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:41.138863    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:24:45.647900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:45.647867    4917 scope.go:122] "RemoveContainer" containerID="93cc485fd465d7959805606cd9780654f7b452d7dda6a610b3102c4a79f55c52"
Oct 30 20:24:45.659815 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:45.659789    4917 scope.go:122] "RemoveContainer" containerID="6c893608a542f51b8ceba48f6b6907f30c72580804f12b010ae6c1631e03e6e9"
Oct 30 20:24:45.668914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:45.668895    4917 scope.go:122] "RemoveContainer" containerID="4198e9deedf216f44fcc2e75fae3be4b4f812589213be9af14b1066107872c84"
Oct 30 20:24:53.305802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:53.305746    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"]
Oct 30 20:24:53.355601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:53.355567    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f75272-gr7fk"]
Oct 30 20:24:54.002056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:54.002022    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416fd293-cf90-4b96-842d-a56d279807bc" path="/var/lib/kubelet/pods/416fd293-cf90-4b96-842d-a56d279807bc/volumes"
Oct 30 20:24:54.576504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:54.576466    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"]
Oct 30 20:24:54.626915 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:54.626884    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebb42-d62lp"]
Oct 30 20:24:56.001856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:24:56.001827    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143d16d4-48d0-4257-9d59-3f475da2c68d" path="/var/lib/kubelet/pods/143d16d4-48d0-4257-9d59-3f475da2c68d/volumes"
Oct 30 20:25:00.838424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:00.838384    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"]
Oct 30 20:25:00.838891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:00.838624    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy" containerID="cri-o://13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481" gracePeriod=30
Oct 30 20:25:02.837620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:02.837587    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"]
Oct 30 20:25:02.838162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:02.837895    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy" containerID="cri-o://7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212" gracePeriod=30
Oct 30 20:25:06.097655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.097629    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:25:06.211475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211443    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211512    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211552    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211609    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211656    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211731    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.211920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211746    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211835    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.211920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211871    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.212202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211916    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token\") pod \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\" (UID: \"f1abb2b7-0c76-49f0-8f81-3686caf389cd\") "
Oct 30 20:25:06.212202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.211998    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.212202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212029    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.212411 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212353    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:06.212514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212495    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.212514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212516    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.212719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212531    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.212719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212544    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.212719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.212551    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.214277 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.214251    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.214387 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.214295    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "kube-api-access-6j785". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:06.214452 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.214399    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:06.214527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.214463    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token" pod "f1abb2b7-0c76-49f0-8f81-3686caf389cd" (UID: "f1abb2b7-0c76-49f0-8f81-3686caf389cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:06.347490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.338362    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.347490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.347162    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.347490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.347203    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.347490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.347222    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j785\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-kube-api-access-6j785\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.347490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.347238    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abb2b7-0c76-49f0-8f81-3686caf389cd-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.737843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.737810    4917 generic.go:356] "Generic (PLEG): container finished" podID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerID="13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481" exitCode=0
Oct 30 20:25:06.738034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.737907    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"
Oct 30 20:25:06.738034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.737930    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" event={"ID":"f1abb2b7-0c76-49f0-8f81-3686caf389cd","Type":"ContainerDied","Data":"13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481"}
Oct 30 20:25:06.738034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.737985    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck" event={"ID":"f1abb2b7-0c76-49f0-8f81-3686caf389cd","Type":"ContainerDied","Data":"79d905423e58f596185e94ac6f763637078ae0643f9d6f353ec88ca3e247bbc9"}
Oct 30 20:25:06.738034 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.738006    4917 scope.go:122] "RemoveContainer" containerID="13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481"
Oct 30 20:25:06.756631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.756608    4917 scope.go:122] "RemoveContainer" containerID="13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481"
Oct 30 20:25:06.756978 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:25:06.756953    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481\": container with ID starting with 13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481 not found: ID does not exist" containerID="13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481"
Oct 30 20:25:06.757091 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.756984    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481"} err="failed to get container status \"13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481\": rpc error: code = NotFound desc = could not find container \"13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481\": container with ID starting with 13f6cbf3806cec9db283d69afd016f72de180625d90a2c1147b55f19ba050481 not found: ID does not exist"
Oct 30 20:25:06.908989 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.908957    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"]
Oct 30 20:25:06.959391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:06.959358    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-4wsck"]
Oct 30 20:25:08.004448 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.004412    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" path="/var/lib/kubelet/pods/f1abb2b7-0c76-49f0-8f81-3686caf389cd/volumes"
Oct 30 20:25:08.105137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.105080    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:25:08.167711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167679    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.167902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167724    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.167902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167814    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.167902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167870    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.167902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167899    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.168098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167928    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.168098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167953    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.168098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.167980    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.168098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168013    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data\") pod \"d641ab76-a95f-4043-8e38-d6a7b8323900\" (UID: \"d641ab76-a95f-4043-8e38-d6a7b8323900\") "
Oct 30 20:25:08.168406 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168174    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:08.168406 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168215    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:08.168406 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168401    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:08.168567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168495    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:08.168567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168523    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:08.168655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168588    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d641ab76-a95f-4043-8e38-d6a7b8323900-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.168655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168607    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.168655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.168641    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.170199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.170178    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:08.170299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.170246    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:08.170508 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.170479    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "kube-api-access-5wj9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:08.170604 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.170516    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token" pod "d641ab76-a95f-4043-8e38-d6a7b8323900" (UID: "d641ab76-a95f-4043-8e38-d6a7b8323900"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:08.269830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.269795    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wj9z\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-kube-api-access-5wj9z\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.269830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.269827    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.269830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.269837    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.270053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.269845    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.270053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.269854    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.270053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.269862    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d641ab76-a95f-4043-8e38-d6a7b8323900-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:08.754040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.754005    4917 generic.go:356] "Generic (PLEG): container finished" podID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerID="7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212" exitCode=0
Oct 30 20:25:08.754197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.754054    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"
Oct 30 20:25:08.754197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.754094    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" event={"ID":"d641ab76-a95f-4043-8e38-d6a7b8323900","Type":"ContainerDied","Data":"7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212"}
Oct 30 20:25:08.754197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.754152    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq" event={"ID":"d641ab76-a95f-4043-8e38-d6a7b8323900","Type":"ContainerDied","Data":"ea5f0c44031f6aba9d176763ddb87e83154221c8d955038de1a5bc84c207c6cc"}
Oct 30 20:25:08.754197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.754175    4917 scope.go:122] "RemoveContainer" containerID="7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212"
Oct 30 20:25:08.765023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.765004    4917 scope.go:122] "RemoveContainer" containerID="7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212"
Oct 30 20:25:08.765282 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:25:08.765262    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212\": container with ID starting with 7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212 not found: ID does not exist" containerID="7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212"
Oct 30 20:25:08.765360 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.765291    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212"} err="failed to get container status \"7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212\": rpc error: code = NotFound desc = could not find container \"7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212\": container with ID starting with 7eb1e557bf3c90a051bb503dcf1feda54017089d878f15760f6d83ca3ecf4212 not found: ID does not exist"
Oct 30 20:25:08.975021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:08.974983    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"]
Oct 30 20:25:09.025069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:09.024991    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fh2qq"]
Oct 30 20:25:09.681234 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:25:09.681187    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:25:10.002754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:10.002679    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" path="/var/lib/kubelet/pods/d641ab76-a95f-4043-8e38-d6a7b8323900/volumes"
Oct 30 20:25:17.501908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:17.501874    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:25:30.056995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:30.056955    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"]
Oct 30 20:25:30.106173 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:30.106139    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bb79f38-ckl87"]
Oct 30 20:25:32.002173 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:32.002142    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e156de-2914-49d7-8a96-dc624914ea68" path="/var/lib/kubelet/pods/60e156de-2914-49d7-8a96-dc624914ea68/volumes"
Oct 30 20:25:39.081421 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:39.081381    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:25:39.680263 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:25:39.680221    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:25:45.724226 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:45.724197    4917 scope.go:122] "RemoveContainer" containerID="2c71f141bb6beb16c52a8b83ffcaea4e4040c3cd11fcdaa507a61bb78941348c"
Oct 30 20:25:45.734115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:45.733826    4917 scope.go:122] "RemoveContainer" containerID="533399de30d1a1fdc4dcac90f93ab75a24c0a7735ebe1dd89d3fdd49f1f2b3f2"
Oct 30 20:25:45.742623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:45.742604    4917 scope.go:122] "RemoveContainer" containerID="9d66280ea606d0028fb98e5e7563f02683d08f12abe3c0f0a07ad636b180aa7f"
Oct 30 20:25:45.751610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:45.751592    4917 scope.go:122] "RemoveContainer" containerID="49c199d91de1d424043119f1e6efed2faf6e3b9c7338ee86d31666c73572ca55"
Oct 30 20:25:46.014603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.014529    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"]
Oct 30 20:25:46.014966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.014954    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy"
Oct 30 20:25:46.015019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.014967    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy"
Oct 30 20:25:46.015019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.014984    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy"
Oct 30 20:25:46.015019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.014989    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy"
Oct 30 20:25:46.015019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015017    4917 cpu_manager.go:414] "containerMap: removing container" podUID="d6bb20b0-c159-4221-a734-103a05b35eff" containerName="revoke-keys"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015022    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bb20b0-c159-4221-a734-103a05b35eff" containerName="revoke-keys"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015028    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" containerName="revoke-keys"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015032    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" containerName="revoke-keys"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015107    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d641ab76-a95f-4043-8e38-d6a7b8323900" containerName="istio-proxy"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015122    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bb20b0-c159-4221-a734-103a05b35eff" containerName="revoke-keys"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015127    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" containerName="revoke-keys"
Oct 30 20:25:46.015143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.015135    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1abb2b7-0c76-49f0-8f81-3686caf389cd" containerName="istio-proxy"
Oct 30 20:25:46.019883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.019867    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.076197 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.076168    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:25:46.079509 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.079484    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"]
Oct 30 20:25:46.107717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.107679    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6tzw\" (UniqueName: \"kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.107884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.107761    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.108021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.108001    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.126359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.126334    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:25:46.126492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.126361    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:25:46.126492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.126436    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:25:46.208575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.208543    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6tzw\" (UniqueName: \"kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.208738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.208583    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.208738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.208649    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.209440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.209422    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.211046 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.211029    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.270442 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.270382    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6tzw\" (UniqueName: \"kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw\") pod \"maas-api-revoke-sub-d234dbf7-bbfc7\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.341087 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.341054    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:46.719436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:46.719409    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"]
Oct 30 20:25:46.720224 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:25:46.720198    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef098152_3a2b_437d_8633_dbe02247f55f.slice/crio-58c2f1b32cb45834101a3ab7e7f619bde3c84c262749df2147d4a5bba33e1530 WatchSource:0}: Error finding container 58c2f1b32cb45834101a3ab7e7f619bde3c84c262749df2147d4a5bba33e1530: Status 404 returned error can't find the container with id 58c2f1b32cb45834101a3ab7e7f619bde3c84c262749df2147d4a5bba33e1530
Oct 30 20:25:47.009314 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:47.009231    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7" event={"ID":"ef098152-3a2b-437d-8633-dbe02247f55f","Type":"ContainerStarted","Data":"58c2f1b32cb45834101a3ab7e7f619bde3c84c262749df2147d4a5bba33e1530"}
Oct 30 20:25:48.016632 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:48.016601    4917 generic.go:356] "Generic (PLEG): container finished" podID="ef098152-3a2b-437d-8633-dbe02247f55f" containerID="6bbff44922d009ba381ec31c7bd06c671ab37ed6a9af920f6075e9ceacb57871" exitCode=0
Oct 30 20:25:48.017056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:48.016644    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7" event={"ID":"ef098152-3a2b-437d-8633-dbe02247f55f","Type":"ContainerDied","Data":"6bbff44922d009ba381ec31c7bd06c671ab37ed6a9af920f6075e9ceacb57871"}
Oct 30 20:25:49.157385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.157361    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:49.241619 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.241588    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca\") pod \"ef098152-3a2b-437d-8633-dbe02247f55f\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") "
Oct 30 20:25:49.241833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.241709    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw\" (UniqueName: \"kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw\") pod \"ef098152-3a2b-437d-8633-dbe02247f55f\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") "
Oct 30 20:25:49.241833 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.241755    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca\") pod \"ef098152-3a2b-437d-8633-dbe02247f55f\" (UID: \"ef098152-3a2b-437d-8633-dbe02247f55f\") "
Oct 30 20:25:49.242068 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.242043    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca" pod "ef098152-3a2b-437d-8633-dbe02247f55f" (UID: "ef098152-3a2b-437d-8633-dbe02247f55f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:49.242248 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.242230    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:49.243961 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.243929    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw" pod "ef098152-3a2b-437d-8633-dbe02247f55f" (UID: "ef098152-3a2b-437d-8633-dbe02247f55f"). InnerVolumeSpecName "kube-api-access-g6tzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:49.244092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.243969    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca" pod "ef098152-3a2b-437d-8633-dbe02247f55f" (UID: "ef098152-3a2b-437d-8633-dbe02247f55f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:49.342884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.342762    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6tzw\" (UniqueName: \"kubernetes.io/projected/ef098152-3a2b-437d-8633-dbe02247f55f-kube-api-access-g6tzw\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:49.342884 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:49.342835    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef098152-3a2b-437d-8633-dbe02247f55f-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:50.031643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:50.031616    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"
Oct 30 20:25:50.031848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:50.031660    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7" event={"ID":"ef098152-3a2b-437d-8633-dbe02247f55f","Type":"ContainerDied","Data":"58c2f1b32cb45834101a3ab7e7f619bde3c84c262749df2147d4a5bba33e1530"}
Oct 30 20:25:50.031848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:25:50.031703    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c2f1b32cb45834101a3ab7e7f619bde3c84c262749df2147d4a5bba33e1530"
Oct 30 20:26:03.059389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:03.059347    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"]
Oct 30 20:26:03.088310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:03.088267    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7256b4-5pflb"]
Oct 30 20:26:04.003686 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:04.003654    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1aa486-e16c-4601-a017-2899c7027b79" path="/var/lib/kubelet/pods/7f1aa486-e16c-4601-a017-2899c7027b79/volumes"
Oct 30 20:26:09.680012 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:26:09.679967    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:26:10.928001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.927966    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"]
Oct 30 20:26:10.928660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.928636    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ef098152-3a2b-437d-8633-dbe02247f55f" containerName="revoke-keys"
Oct 30 20:26:10.928764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.928661    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef098152-3a2b-437d-8633-dbe02247f55f" containerName="revoke-keys"
Oct 30 20:26:10.928880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.928803    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef098152-3a2b-437d-8633-dbe02247f55f" containerName="revoke-keys"
Oct 30 20:26:10.932049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.932028    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:10.983174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.983144    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:10.983327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.983202    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:10.983384 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.983350    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gzk\" (UniqueName: \"kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:10.990408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:10.990378    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"]
Oct 30 20:26:11.037066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.037008    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:26:11.037066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.037043    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:26:11.037066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.037054    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:26:11.037323 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.037156    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:26:11.084394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.084365    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gzk\" (UniqueName: \"kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.084563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.084472    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.084563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.084506    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.085306 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.085280    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.086998 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.086977    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.147061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.147027    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gzk\" (UniqueName: \"kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk\") pod \"maas-api-revoke-sub-fc3ac03c-gcllq\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.245361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.245284    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:11.613940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.613862    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:26:11.627032 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:11.627003    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"]
Oct 30 20:26:11.627760 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:26:11.627729    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf11604b_818d_480b_b2de_ef509defff71.slice/crio-5ece50a127016db78e029a74f2d84d8998072d021ca008c269858d961dde46b6 WatchSource:0}: Error finding container 5ece50a127016db78e029a74f2d84d8998072d021ca008c269858d961dde46b6: Status 404 returned error can't find the container with id 5ece50a127016db78e029a74f2d84d8998072d021ca008c269858d961dde46b6
Oct 30 20:26:12.182694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:12.182651    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq" event={"ID":"cf11604b-818d-480b-b2de-ef509defff71","Type":"ContainerStarted","Data":"61f07b3457204088348b7c5d45e073d847505838157bc352700bc6a99e163ffc"}
Oct 30 20:26:12.183201 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:12.182714    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq" event={"ID":"cf11604b-818d-480b-b2de-ef509defff71","Type":"ContainerStarted","Data":"5ece50a127016db78e029a74f2d84d8998072d021ca008c269858d961dde46b6"}
Oct 30 20:26:12.317004 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:12.315974    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq" podStartSLOduration=2.315954429 podStartE2EDuration="2.315954429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:12.312821979 +0000 UTC m=+2668.477223623" watchObservedRunningTime="2026-10-30 20:26:12.315954429 +0000 UTC m=+2668.480356070"
Oct 30 20:26:13.192250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:13.192213    4917 generic.go:356] "Generic (PLEG): container finished" podID="cf11604b-818d-480b-b2de-ef509defff71" containerID="61f07b3457204088348b7c5d45e073d847505838157bc352700bc6a99e163ffc" exitCode=0
Oct 30 20:26:13.192704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:13.192357    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq" event={"ID":"cf11604b-818d-480b-b2de-ef509defff71","Type":"ContainerDied","Data":"61f07b3457204088348b7c5d45e073d847505838157bc352700bc6a99e163ffc"}
Oct 30 20:26:14.355074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.355041    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:14.433486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.433457    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk\" (UniqueName: \"kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk\") pod \"cf11604b-818d-480b-b2de-ef509defff71\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") "
Oct 30 20:26:14.433692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.433508    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca\") pod \"cf11604b-818d-480b-b2de-ef509defff71\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") "
Oct 30 20:26:14.433692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.433588    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca\") pod \"cf11604b-818d-480b-b2de-ef509defff71\" (UID: \"cf11604b-818d-480b-b2de-ef509defff71\") "
Oct 30 20:26:14.434061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.434029    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca" pod "cf11604b-818d-480b-b2de-ef509defff71" (UID: "cf11604b-818d-480b-b2de-ef509defff71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:14.435662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.435643    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca" pod "cf11604b-818d-480b-b2de-ef509defff71" (UID: "cf11604b-818d-480b-b2de-ef509defff71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:14.435753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.435680    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk" pod "cf11604b-818d-480b-b2de-ef509defff71" (UID: "cf11604b-818d-480b-b2de-ef509defff71"). InnerVolumeSpecName "kube-api-access-22gzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:14.535457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.535377    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gzk\" (UniqueName: \"kubernetes.io/projected/cf11604b-818d-480b-b2de-ef509defff71-kube-api-access-22gzk\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:14.535457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.535407    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf11604b-818d-480b-b2de-ef509defff71-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:14.535457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:14.535417    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf11604b-818d-480b-b2de-ef509defff71-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:15.207968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:15.207936    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"
Oct 30 20:26:15.208148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:15.207970    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq" event={"ID":"cf11604b-818d-480b-b2de-ef509defff71","Type":"ContainerDied","Data":"5ece50a127016db78e029a74f2d84d8998072d021ca008c269858d961dde46b6"}
Oct 30 20:26:15.208148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:15.208017    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ece50a127016db78e029a74f2d84d8998072d021ca008c269858d961dde46b6"
Oct 30 20:26:37.741280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:37.741240    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"]
Oct 30 20:26:37.741926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:37.741474    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" podUID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" containerName="discovery" containerID="cri-o://d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d" gracePeriod=30
Oct 30 20:26:38.003633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.003567    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:26:38.194123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194082    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.194322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194132    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.194322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194171    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.194322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194224    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.194513 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194309    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.194640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194609    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:38.194761 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194738    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.194916 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.194897    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts\") pod \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\" (UID: \"6ba833ad-696a-47a2-acb7-7b8c8aa84278\") "
Oct 30 20:26:38.195299 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.195280    4917 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-ca-configmap\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.196968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.196933    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:38.196968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.196950    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:38.197115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.197045    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:38.197484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.197462    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "kube-api-access-mr8h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:38.197484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.197471    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:38.197603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.197519    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts" pod "6ba833ad-696a-47a2-acb7-7b8c8aa84278" (UID: "6ba833ad-696a-47a2-acb7-7b8c8aa84278"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:38.296759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.296677    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr8h4\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-kube-api-access-mr8h4\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.296759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.296708    4917 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-cacerts\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.296759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.296718    4917 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba833ad-696a-47a2-acb7-7b8c8aa84278-local-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.296759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.296728    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.296759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.296737    4917 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-csr-dns-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.296759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.296746    4917 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ba833ad-696a-47a2-acb7-7b8c8aa84278-istio-kubeconfig\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:38.354462 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.354431    4917 generic.go:356] "Generic (PLEG): container finished" podID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" containerID="d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d" exitCode=0
Oct 30 20:26:38.354640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.354495    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"
Oct 30 20:26:38.354640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.354509    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" event={"ID":"6ba833ad-696a-47a2-acb7-7b8c8aa84278","Type":"ContainerDied","Data":"d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d"}
Oct 30 20:26:38.354640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.354560    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6" event={"ID":"6ba833ad-696a-47a2-acb7-7b8c8aa84278","Type":"ContainerDied","Data":"81db0796fcb5f8ee996ffc04a0cd9daf81e70ec813b9359ac048ecfc6c5e1db9"}
Oct 30 20:26:38.354640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.354582    4917 scope.go:122] "RemoveContainer" containerID="d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d"
Oct 30 20:26:38.365808 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.365788    4917 scope.go:122] "RemoveContainer" containerID="d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d"
Oct 30 20:26:38.366050 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:26:38.366032    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d\": container with ID starting with d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d not found: ID does not exist" containerID="d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d"
Oct 30 20:26:38.366114 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.366058    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d"} err="failed to get container status \"d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d\": rpc error: code = NotFound desc = could not find container \"d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d\": container with ID starting with d5b57361bc1fe51217a639d14fa8995b385326becd1b2c1d6f38b38c83408c5d not found: ID does not exist"
Oct 30 20:26:38.524287 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.524248    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"]
Oct 30 20:26:38.574903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:38.574831    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lh2q6"]
Oct 30 20:26:39.687580 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:26:39.687513    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:26:40.002712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:40.002634    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" path="/var/lib/kubelet/pods/6ba833ad-696a-47a2-acb7-7b8c8aa84278/volumes"
Oct 30 20:26:44.386975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:44.386949    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:26:44.445681 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:44.445652    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:26:45.798194 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:45.798160    4917 scope.go:122] "RemoveContainer" containerID="4f8215345b040b117fe26f2b5f148bdaffaf0d9f63d322f272dd3283de5fffa8"
Oct 30 20:26:45.807573 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:45.807552    4917 scope.go:122] "RemoveContainer" containerID="433af2dfe56a4c1a23e9001b6ebadf0d624637ec9470329150305e8ad1802254"
Oct 30 20:26:45.816252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:45.816223    4917 scope.go:122] "RemoveContainer" containerID="4af13676097c3b4f6fb5d5a48992fe39dfd5eb168ed483f27584c32b6a8f6adc"
Oct 30 20:26:45.824934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:45.824916    4917 scope.go:122] "RemoveContainer" containerID="ac2b8095a6606563d069e72c878a7ba5c8750a19be81391f5f46b6f6d66399cb"
Oct 30 20:26:46.490419 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.490384    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"]
Oct 30 20:26:46.490927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.490908    4917 cpu_manager.go:414] "containerMap: removing container" podUID="cf11604b-818d-480b-b2de-ef509defff71" containerName="revoke-keys"
Oct 30 20:26:46.491047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.490928    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf11604b-818d-480b-b2de-ef509defff71" containerName="revoke-keys"
Oct 30 20:26:46.491047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.490945    4917 cpu_manager.go:414] "containerMap: removing container" podUID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" containerName="discovery"
Oct 30 20:26:46.491047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.490953    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" containerName="discovery"
Oct 30 20:26:46.491220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.491115    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf11604b-818d-480b-b2de-ef509defff71" containerName="revoke-keys"
Oct 30 20:26:46.491220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.491129    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba833ad-696a-47a2-acb7-7b8c8aa84278" containerName="discovery"
Oct 30 20:26:46.495968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.495942    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.581262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.581230    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.581457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.581364    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.581457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.581395    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdspc\" (UniqueName: \"kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.601978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.601937    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:26:46.601978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.601968    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:26:46.602162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.602056    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:26:46.602162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.602059    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:26:46.605189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.605169    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"]
Oct 30 20:26:46.682721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.682685    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.682920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.682727    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdspc\" (UniqueName: \"kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.682920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.682885    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.683537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.683517    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.685150 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.685126    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.744669 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.744607    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdspc\" (UniqueName: \"kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc\") pod \"maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:46.810666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:46.810636    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:47.193303 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:47.193273    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"]
Oct 30 20:26:47.194636 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:26:47.194604    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269d23c7_b5bd_4f9b_b0ba_0f255011c52d.slice/crio-c9165c4e9f1be504db0ad332f89c7b361e1368b4b34573d5629fbf15a42238b0 WatchSource:0}: Error finding container c9165c4e9f1be504db0ad332f89c7b361e1368b4b34573d5629fbf15a42238b0: Status 404 returned error can't find the container with id c9165c4e9f1be504db0ad332f89c7b361e1368b4b34573d5629fbf15a42238b0
Oct 30 20:26:47.412609 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:47.412575    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r" event={"ID":"269d23c7-b5bd-4f9b-b0ba-0f255011c52d","Type":"ContainerStarted","Data":"c9165c4e9f1be504db0ad332f89c7b361e1368b4b34573d5629fbf15a42238b0"}
Oct 30 20:26:48.420518 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:48.420418    4917 generic.go:356] "Generic (PLEG): container finished" podID="269d23c7-b5bd-4f9b-b0ba-0f255011c52d" containerID="dcbecc95e7c95e4ef2eb74c6c9ac34201720a7dedc7a707fb7ed56717e62e099" exitCode=0
Oct 30 20:26:48.420518 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:48.420468    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r" event={"ID":"269d23c7-b5bd-4f9b-b0ba-0f255011c52d","Type":"ContainerDied","Data":"dcbecc95e7c95e4ef2eb74c6c9ac34201720a7dedc7a707fb7ed56717e62e099"}
Oct 30 20:26:49.335309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.335272    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:26:49.570994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.570968    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:49.613490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.613414    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca\") pod \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") "
Oct 30 20:26:49.613490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.613458    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc\" (UniqueName: \"kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc\") pod \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") "
Oct 30 20:26:49.613692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.613588    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca\") pod \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\" (UID: \"269d23c7-b5bd-4f9b-b0ba-0f255011c52d\") "
Oct 30 20:26:49.613889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.613862    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca" pod "269d23c7-b5bd-4f9b-b0ba-0f255011c52d" (UID: "269d23c7-b5bd-4f9b-b0ba-0f255011c52d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:49.614108 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.614091    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.615594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.615573    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca" pod "269d23c7-b5bd-4f9b-b0ba-0f255011c52d" (UID: "269d23c7-b5bd-4f9b-b0ba-0f255011c52d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:49.615677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.615658    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc" pod "269d23c7-b5bd-4f9b-b0ba-0f255011c52d" (UID: "269d23c7-b5bd-4f9b-b0ba-0f255011c52d"). InnerVolumeSpecName "kube-api-access-sdspc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:49.715474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.715440    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdspc\" (UniqueName: \"kubernetes.io/projected/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-kube-api-access-sdspc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.715474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:49.715468    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269d23c7-b5bd-4f9b-b0ba-0f255011c52d-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:50.437630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:50.437582    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r" event={"ID":"269d23c7-b5bd-4f9b-b0ba-0f255011c52d","Type":"ContainerDied","Data":"c9165c4e9f1be504db0ad332f89c7b361e1368b4b34573d5629fbf15a42238b0"}
Oct 30 20:26:50.437879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:50.437643    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"
Oct 30 20:26:50.437879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:50.437650    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9165c4e9f1be504db0ad332f89c7b361e1368b4b34573d5629fbf15a42238b0"
Oct 30 20:26:57.537755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.537719    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"]
Oct 30 20:26:57.538208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.538193    4917 cpu_manager.go:414] "containerMap: removing container" podUID="269d23c7-b5bd-4f9b-b0ba-0f255011c52d" containerName="revoke-keys"
Oct 30 20:26:57.538255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.538208    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269d23c7-b5bd-4f9b-b0ba-0f255011c52d" containerName="revoke-keys"
Oct 30 20:26:57.538302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.538293    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="269d23c7-b5bd-4f9b-b0ba-0f255011c52d" containerName="revoke-keys"
Oct 30 20:26:57.541563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.541546    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.597374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.597335    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-72db69\"/\"default-dockercfg-gklb2\""
Oct 30 20:26:57.599879 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.599846    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.600047 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.600022    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.600126 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.600083    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.600185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.600170    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstbx\" (UniqueName: \"kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.600249 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.600205    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.600344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.600317    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.600610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.600594    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"]
Oct 30 20:26:57.647841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.647811    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-72db69\"/\"test-model-72db69-kserve-self-signed-certs\""
Oct 30 20:26:57.647841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.647822    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-72db69\"/\"openshift-service-ca.crt\""
Oct 30 20:26:57.648036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.647862    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-72db69\"/\"kube-root-ca.crt\""
Oct 30 20:26:57.701203 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701159    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.701441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701298    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.701441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701346    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.701441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701382    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstbx\" (UniqueName: \"kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.701441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701407    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.701631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701455    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.701682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.701663    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.702119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.702073    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.702553 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.702525    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.704223 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.704199    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.704966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.704945    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.763386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.763355    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstbx\" (UniqueName: \"kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx\") pod \"test-model-72db69-kserve-6c4df7c859-wcgkv\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") " pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:57.855420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:57.855329    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:26:58.245305 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:58.245276    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"]
Oct 30 20:26:58.246171 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:26:58.246136    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40593e05_8672_4df6_9e84_a64da221f298.slice/crio-4fc15a40c69273261aca2aec0cdea4b706e3b50e7bd04937fd883dd3a6f00589 WatchSource:0}: Error finding container 4fc15a40c69273261aca2aec0cdea4b706e3b50e7bd04937fd883dd3a6f00589: Status 404 returned error can't find the container with id 4fc15a40c69273261aca2aec0cdea4b706e3b50e7bd04937fd883dd3a6f00589
Oct 30 20:26:58.488092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:58.488049    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv" event={"ID":"40593e05-8672-4df6-9e84-a64da221f298","Type":"ContainerStarted","Data":"e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239"}
Oct 30 20:26:58.488092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:58.488098    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv" event={"ID":"40593e05-8672-4df6-9e84-a64da221f298","Type":"ContainerStarted","Data":"4fc15a40c69273261aca2aec0cdea4b706e3b50e7bd04937fd883dd3a6f00589"}
Oct 30 20:26:58.609693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:58.609592    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv" podStartSLOduration=1.609577373 podStartE2EDuration="1.609577373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:58.607499755 +0000 UTC m=+2714.771901398" watchObservedRunningTime="2026-10-30 20:26:58.609577373 +0000 UTC m=+2714.773979014"
Oct 30 20:26:58.856109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:26:58.856078    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:27:06.094471 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:06.094430    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"]
Oct 30 20:27:06.143312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:06.143270    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0051fb-7l44r"]
Oct 30 20:27:07.855672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:07.855641    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:27:07.868446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:07.868417    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:27:08.004959 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:08.004924    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269d23c7-b5bd-4f9b-b0ba-0f255011c52d" path="/var/lib/kubelet/pods/269d23c7-b5bd-4f9b-b0ba-0f255011c52d/volumes"
Oct 30 20:27:08.566479 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:08.566448    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:27:09.679445 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:27:09.679410    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:27:11.064784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:11.064736    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"]
Oct 30 20:27:11.115096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:11.115051    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd054f2-gsnxp"]
Oct 30 20:27:12.002227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:12.002197    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd32c29-63ce-4952-8e4d-e9b4a909e7eb" path="/var/lib/kubelet/pods/6cd32c29-63ce-4952-8e4d-e9b4a909e7eb/volumes"
Oct 30 20:27:14.254938 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.254898    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"]
Oct 30 20:27:14.261445 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.261414    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.300363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.300333    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.300525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.300371    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.300525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.300468    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zjr\" (UniqueName: \"kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.314022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.313996    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"]
Oct 30 20:27:14.360982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.360950    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:27:14.361169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.361099    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:27:14.365378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.365357    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:27:14.365378 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.365374    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:27:14.401752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.401722    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.401930 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.401758    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.401930 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.401832    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zjr\" (UniqueName: \"kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.402444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.402424    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.404238 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.404219    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.476867 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.476834    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zjr\" (UniqueName: \"kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr\") pod \"maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.576332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.576239    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:14.950745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:14.950720    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"]
Oct 30 20:27:14.953628 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:27:14.953593    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe1c2b0_4d8e_42cf_994b_f48df2d2d2f4.slice/crio-f246f60e9ad5acc610e86d83137e860b2f1797fa3c1166799342986d058f678c WatchSource:0}: Error finding container f246f60e9ad5acc610e86d83137e860b2f1797fa3c1166799342986d058f678c: Status 404 returned error can't find the container with id f246f60e9ad5acc610e86d83137e860b2f1797fa3c1166799342986d058f678c
Oct 30 20:27:15.615456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:15.615379    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx" event={"ID":"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4","Type":"ContainerStarted","Data":"9a8010565ba9bfd4ef8bb543b29d0a37da796b32570332401627d67fafc8d8d9"}
Oct 30 20:27:15.615456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:15.615430    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx" event={"ID":"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4","Type":"ContainerStarted","Data":"f246f60e9ad5acc610e86d83137e860b2f1797fa3c1166799342986d058f678c"}
Oct 30 20:27:15.732786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:15.732720    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx" podStartSLOduration=1.732705785 podStartE2EDuration="1.732705785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:15.73146298 +0000 UTC m=+2731.895864644" watchObservedRunningTime="2026-10-30 20:27:15.732705785 +0000 UTC m=+2731.897107435"
Oct 30 20:27:16.624405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:16.624373    4917 generic.go:356] "Generic (PLEG): container finished" podID="ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" containerID="9a8010565ba9bfd4ef8bb543b29d0a37da796b32570332401627d67fafc8d8d9" exitCode=0
Oct 30 20:27:16.624843 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:16.624501    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx" event={"ID":"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4","Type":"ContainerDied","Data":"9a8010565ba9bfd4ef8bb543b29d0a37da796b32570332401627d67fafc8d8d9"}
Oct 30 20:27:17.784118 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.784094    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:17.836106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.836061    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca\") pod \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") "
Oct 30 20:27:17.836314 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.836226    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr\" (UniqueName: \"kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr\") pod \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") "
Oct 30 20:27:17.836314 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.836299    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca\") pod \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\" (UID: \"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4\") "
Oct 30 20:27:17.836496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.836467    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca" pod "ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" (UID: "ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:17.836676 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.836657    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:17.838367 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.838336    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca" pod "ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" (UID: "ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:17.838504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.838417    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr" pod "ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" (UID: "ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4"). InnerVolumeSpecName "kube-api-access-q6zjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:17.938137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.938104    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:17.938137 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:17.938135    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zjr\" (UniqueName: \"kubernetes.io/projected/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4-kube-api-access-q6zjr\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:18.638454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:18.638417    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx" event={"ID":"ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4","Type":"ContainerDied","Data":"f246f60e9ad5acc610e86d83137e860b2f1797fa3c1166799342986d058f678c"}
Oct 30 20:27:18.638454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:18.638442    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"
Oct 30 20:27:18.638674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:18.638463    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f246f60e9ad5acc610e86d83137e860b2f1797fa3c1166799342986d058f678c"
Oct 30 20:27:30.039628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:30.039549    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:27:39.679354 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:27:39.679315    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:27:41.415215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:41.415173    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"]
Oct 30 20:27:41.415718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:41.415422    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy" containerID="cri-o://868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb" gracePeriod=30
Oct 30 20:27:44.763039 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:44.763003    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"]
Oct 30 20:27:44.763533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:44.763261    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" containerID="cri-o://98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403" gracePeriod=30
Oct 30 20:27:45.866237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:45.866203    4917 scope.go:122] "RemoveContainer" containerID="d61ac67f070c0c759dcde4446cb87053af6f224ca05781c0f0ae43f3b7fe81cc"
Oct 30 20:27:45.875747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:45.875723    4917 scope.go:122] "RemoveContainer" containerID="6b8c2771d4358710096276bf41126b80020c4ba10648d1771aa89a73c54fb99c"
Oct 30 20:27:45.884698 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:45.884677    4917 scope.go:122] "RemoveContainer" containerID="c88c366d8ed5edb5e9203fe62c58b849529a2b50bb2d3ecc8b044b2f3d830357"
Oct 30 20:27:46.685645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.685578    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:27:46.751410 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751382    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751417    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751458    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751498    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751528    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751554    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751576    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.751612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751606    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.752031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751659    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs\") pod \"a705b05f-e0b3-4b7c-bc09-693284568acc\" (UID: \"a705b05f-e0b3-4b7c-bc09-693284568acc\") "
Oct 30 20:27:46.752031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751700    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:46.752031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751903    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:46.752031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.751923    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:46.752254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.752108    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.752254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.752122    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:46.752254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.752128    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.752254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.752149    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:46.752254 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.752165    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.753811 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.753786    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:46.753934 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.753888    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:27:46.754297 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.754265    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:46.754405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.754301    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h" pod "a705b05f-e0b3-4b7c-bc09-693284568acc" (UID: "a705b05f-e0b3-4b7c-bc09-693284568acc"). InnerVolumeSpecName "kube-api-access-dkt8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:46.821786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.821227    4917 generic.go:356] "Generic (PLEG): container finished" podID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerID="868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb" exitCode=0
Oct 30 20:27:46.821786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.821283    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"
Oct 30 20:27:46.821786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.821283    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" event={"ID":"a705b05f-e0b3-4b7c-bc09-693284568acc","Type":"ContainerDied","Data":"868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb"}
Oct 30 20:27:46.821786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.821397    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf" event={"ID":"a705b05f-e0b3-4b7c-bc09-693284568acc","Type":"ContainerDied","Data":"ee90e6a319403921ab8bbc7ba73e9afa02ff904d36ed28f21d7582fc388bc362"}
Oct 30 20:27:46.821786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.821431    4917 scope.go:122] "RemoveContainer" containerID="868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb"
Oct 30 20:27:46.834334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.834307    4917 scope.go:122] "RemoveContainer" containerID="868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb"
Oct 30 20:27:46.834633 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:27:46.834613    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb\": container with ID starting with 868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb not found: ID does not exist" containerID="868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb"
Oct 30 20:27:46.834735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.834641    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb"} err="failed to get container status \"868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb\": rpc error: code = NotFound desc = could not find container \"868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb\": container with ID starting with 868f76b3bb068bc6c68469ae954349f95c2c8758e5e7326718326cd8927fd9cb not found: ID does not exist"
Oct 30 20:27:46.853149 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.853124    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.853149 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.853148    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.853290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.853159    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a705b05f-e0b3-4b7c-bc09-693284568acc-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.853290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.853170    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkt8h\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-kube-api-access-dkt8h\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.853290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.853180    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a705b05f-e0b3-4b7c-bc09-693284568acc-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.853290 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.853191    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a705b05f-e0b3-4b7c-bc09-693284568acc-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:46.989153 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.989061    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"]
Oct 30 20:27:46.993902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:46.993875    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-jgpvf"]
Oct 30 20:27:48.002595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:48.002564    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" path="/var/lib/kubelet/pods/a705b05f-e0b3-4b7c-bc09-693284568acc/volumes"
Oct 30 20:27:49.712331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.712285    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:27:49.712844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.712366    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:27:49.807526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.807495    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"]
Oct 30 20:27:49.807954 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.807940    4917 cpu_manager.go:414] "containerMap: removing container" podUID="ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" containerName="revoke-keys"
Oct 30 20:27:49.808070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.807956    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" containerName="revoke-keys"
Oct 30 20:27:49.808070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.807972    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy"
Oct 30 20:27:49.808070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.807977    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy"
Oct 30 20:27:49.808169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.808071    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" containerName="revoke-keys"
Oct 30 20:27:49.808169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.808079    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a705b05f-e0b3-4b7c-bc09-693284568acc" containerName="istio-proxy"
Oct 30 20:27:49.811363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.811346    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.876901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.876871    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:27:49.877013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.876956    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:27:49.888835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.884669    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.888835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.884744    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.888835 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.884906    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs56k\" (UniqueName: \"kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.922537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.922502    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:27:49.922705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.922586    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:27:49.925598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.925574    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"]
Oct 30 20:27:49.985734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.985642    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs56k\" (UniqueName: \"kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.985734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.985702    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.985950 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.985738    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.986391 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.986370    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:49.988243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:49.988222    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:50.040157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.040128    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:27:50.047594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.047570    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs56k\" (UniqueName: \"kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k\") pod \"maas-api-revoke-sub-ed0000fc-m5q2v\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:50.086460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086427    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086481    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086551    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086597    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086663    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086703    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086740    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.086883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086816    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.087111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086892    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:50.087111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.086905    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data\") pod \"8fa81269-e74c-4900-b895-33b56428061d\" (UID: \"8fa81269-e74c-4900-b895-33b56428061d\") "
Oct 30 20:27:50.087231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087163    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:50.087231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087087    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:50.087377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087359    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:50.087440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087383    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:50.087587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087564    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa81269-e74c-4900-b895-33b56428061d-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.087678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087593    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.087678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087626    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.087678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087652    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.087891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.087679    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.088935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.088908    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:27:50.089472 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.089447    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:50.089575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.089559    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "kube-api-access-f4g5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:50.089651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.089640    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token" pod "8fa81269-e74c-4900-b895-33b56428061d" (UID: "8fa81269-e74c-4900-b895-33b56428061d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:50.161374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.161315    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:50.189088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.189057    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa81269-e74c-4900-b895-33b56428061d-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.189088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.189088    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4g5l\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-kube-api-access-f4g5l\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.189292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.189104    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa81269-e74c-4900-b895-33b56428061d-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.189292 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.189116    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa81269-e74c-4900-b895-33b56428061d-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:50.558890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.558683    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"]
Oct 30 20:27:50.562483 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:27:50.562442    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670fdaef_751f_488c_afb7_2f7ab4e81153.slice/crio-97912b227843fc507125ea7ff9771c9cd4f1193df59242097647b83aa9339633 WatchSource:0}: Error finding container 97912b227843fc507125ea7ff9771c9cd4f1193df59242097647b83aa9339633: Status 404 returned error can't find the container with id 97912b227843fc507125ea7ff9771c9cd4f1193df59242097647b83aa9339633
Oct 30 20:27:50.859311 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.859280    4917 generic.go:356] "Generic (PLEG): container finished" podID="8fa81269-e74c-4900-b895-33b56428061d" containerID="98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403" exitCode=0
Oct 30 20:27:50.859646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.859395    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"
Oct 30 20:27:50.859646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.859391    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" event={"ID":"8fa81269-e74c-4900-b895-33b56428061d","Type":"ContainerDied","Data":"98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403"}
Oct 30 20:27:50.859646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.859554    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs" event={"ID":"8fa81269-e74c-4900-b895-33b56428061d","Type":"ContainerDied","Data":"303b35a8f94aa9729dececb34fbf13412754bb21e2d64d514ca5c6eaaa708db0"}
Oct 30 20:27:50.859646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.859584    4917 scope.go:122] "RemoveContainer" containerID="98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403"
Oct 30 20:27:50.861429 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.861399    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v" event={"ID":"670fdaef-751f-488c-afb7-2f7ab4e81153","Type":"ContainerStarted","Data":"97912b227843fc507125ea7ff9771c9cd4f1193df59242097647b83aa9339633"}
Oct 30 20:27:50.869783 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.869748    4917 scope.go:122] "RemoveContainer" containerID="98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403"
Oct 30 20:27:50.870076 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:27:50.870058    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403\": container with ID starting with 98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403 not found: ID does not exist" containerID="98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403"
Oct 30 20:27:50.870127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:50.870085    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403"} err="failed to get container status \"98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403\": rpc error: code = NotFound desc = could not find container \"98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403\": container with ID starting with 98870ec54862aaec0f51ea3b3526405924dacb7c9f832885b63a9f38a2e25403 not found: ID does not exist"
Oct 30 20:27:51.027371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:51.027335    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"]
Oct 30 20:27:51.032366 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:51.032341    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bmtzs"]
Oct 30 20:27:51.870522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:51.870437    4917 generic.go:356] "Generic (PLEG): container finished" podID="670fdaef-751f-488c-afb7-2f7ab4e81153" containerID="b74dcc8d7006078191d6d65bb57e5f6484f5ededdb68ebb40b133414c3951556" exitCode=0
Oct 30 20:27:51.870522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:51.870478    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v" event={"ID":"670fdaef-751f-488c-afb7-2f7ab4e81153","Type":"ContainerDied","Data":"b74dcc8d7006078191d6d65bb57e5f6484f5ededdb68ebb40b133414c3951556"}
Oct 30 20:27:52.002241 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:52.002207    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa81269-e74c-4900-b895-33b56428061d" path="/var/lib/kubelet/pods/8fa81269-e74c-4900-b895-33b56428061d/volumes"
Oct 30 20:27:53.017817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.017748    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:53.122530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.122500    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca\") pod \"670fdaef-751f-488c-afb7-2f7ab4e81153\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") "
Oct 30 20:27:53.122677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.122573    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k\" (UniqueName: \"kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k\") pod \"670fdaef-751f-488c-afb7-2f7ab4e81153\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") "
Oct 30 20:27:53.122751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.122726    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca\") pod \"670fdaef-751f-488c-afb7-2f7ab4e81153\" (UID: \"670fdaef-751f-488c-afb7-2f7ab4e81153\") "
Oct 30 20:27:53.123084 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.123055    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca" pod "670fdaef-751f-488c-afb7-2f7ab4e81153" (UID: "670fdaef-751f-488c-afb7-2f7ab4e81153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:53.124801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.124759    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca" pod "670fdaef-751f-488c-afb7-2f7ab4e81153" (UID: "670fdaef-751f-488c-afb7-2f7ab4e81153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:53.124905 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.124838    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k" pod "670fdaef-751f-488c-afb7-2f7ab4e81153" (UID: "670fdaef-751f-488c-afb7-2f7ab4e81153"). InnerVolumeSpecName "kube-api-access-cs56k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:53.224407 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.224370    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:53.224407 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.224403    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs56k\" (UniqueName: \"kubernetes.io/projected/670fdaef-751f-488c-afb7-2f7ab4e81153-kube-api-access-cs56k\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:53.224620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.224418    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670fdaef-751f-488c-afb7-2f7ab4e81153-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:53.561271 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.561240    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"]
Oct 30 20:27:53.564325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.564303    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2386c-x5wfx"]
Oct 30 20:27:53.884952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.884872    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v" event={"ID":"670fdaef-751f-488c-afb7-2f7ab4e81153","Type":"ContainerDied","Data":"97912b227843fc507125ea7ff9771c9cd4f1193df59242097647b83aa9339633"}
Oct 30 20:27:53.884952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.884922    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97912b227843fc507125ea7ff9771c9cd4f1193df59242097647b83aa9339633"
Oct 30 20:27:53.884952 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:53.884891    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"
Oct 30 20:27:54.001761 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:54.001718    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4" path="/var/lib/kubelet/pods/ffe1c2b0-4d8e-42cf-994b-f48df2d2d2f4/volumes"
Oct 30 20:27:56.416365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:56.416328    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"]
Oct 30 20:27:56.416824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:27:56.416537    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy" containerID="cri-o://de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85" gracePeriod=30
Oct 30 20:28:01.006672 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.006635    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:28:01.665927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.665901    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:28:01.813524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813435    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.813524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813498    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.813784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813565    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.813784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813597    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.813784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813634    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.813784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813717    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.813784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813749    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.814041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813850    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:01.814041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813903    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:01.814041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813912    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.814041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813965    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:01.814041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.813971    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket\") pod \"e3fd940b-caf5-410b-93e5-45759775b5bf\" (UID: \"e3fd940b-caf5-410b-93e5-45759775b5bf\") "
Oct 30 20:28:01.814326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814150    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:01.814326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814198    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:01.814439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814409    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.814439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814425    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.814439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814434    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.814595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814442    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fd940b-caf5-410b-93e5-45759775b5bf-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.814595 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.814456    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.816122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.816094    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:28:01.816122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.816103    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "kube-api-access-8qrbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:01.816333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.816231    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:01.816333 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.816310    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy" pod "e3fd940b-caf5-410b-93e5-45759775b5bf" (UID: "e3fd940b-caf5-410b-93e5-45759775b5bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:01.915276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.915238    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.915276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.915270    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.915276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.915281    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrbh\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-kube-api-access-8qrbh\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.915520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.915290    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fd940b-caf5-410b-93e5-45759775b5bf-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.938185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.938148    4917 generic.go:356] "Generic (PLEG): container finished" podID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerID="de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85" exitCode=0
Oct 30 20:28:01.938371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.938196    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"
Oct 30 20:28:01.938371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.938203    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" event={"ID":"e3fd940b-caf5-410b-93e5-45759775b5bf","Type":"ContainerDied","Data":"de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85"}
Oct 30 20:28:01.938371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.938258    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk" event={"ID":"e3fd940b-caf5-410b-93e5-45759775b5bf","Type":"ContainerDied","Data":"1514831ddb8a82966090023e90c0a6f40eb7442220c948d0083772d1d0949f7c"}
Oct 30 20:28:01.938371 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.938282    4917 scope.go:122] "RemoveContainer" containerID="de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85"
Oct 30 20:28:01.949453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.949435    4917 scope.go:122] "RemoveContainer" containerID="de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85"
Oct 30 20:28:01.949782 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:28:01.949747    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85\": container with ID starting with de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85 not found: ID does not exist" containerID="de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85"
Oct 30 20:28:01.949844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:01.949793    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85"} err="failed to get container status \"de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85\": rpc error: code = NotFound desc = could not find container \"de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85\": container with ID starting with de77d662345c091e18bd8c230b20f83e5ca0fd43966f5971f97f2a64dce71b85 not found: ID does not exist"
Oct 30 20:28:02.107216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.107132    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"]
Oct 30 20:28:02.112173 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.112141    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-npzwk"]
Oct 30 20:28:02.224439 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224400    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"]
Oct 30 20:28:02.224849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224832    4917 cpu_manager.go:414] "containerMap: removing container" podUID="670fdaef-751f-488c-afb7-2f7ab4e81153" containerName="revoke-keys"
Oct 30 20:28:02.224849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224849    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670fdaef-751f-488c-afb7-2f7ab4e81153" containerName="revoke-keys"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224863    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224869    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224878    4917 cpu_manager.go:414] "containerMap: removing container" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224884    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224957    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="670fdaef-751f-488c-afb7-2f7ab4e81153" containerName="revoke-keys"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224966    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" containerName="istio-proxy"
Oct 30 20:28:02.225015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.224972    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa81269-e74c-4900-b895-33b56428061d" containerName="istio-proxy"
Oct 30 20:28:02.229523 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.229500    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.283638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.283549    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:02.283865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.283634    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:02.283865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.283661    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:02.284017 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.283989    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:28:02.287946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.287924    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"]
Oct 30 20:28:02.318722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.318685    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.318919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.318748    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc45\" (UniqueName: \"kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.318964 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.318926    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.420473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.420372    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.420473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.420417    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc45\" (UniqueName: \"kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.420473 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.420462    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.421123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.421094    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.422960 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.422939    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.482535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.482498    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc45\" (UniqueName: \"kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45\") pod \"maas-api-revoke-keys-e2e-worker-w0-779921-z9dks\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.542550 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.542518    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:02.929618 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.929589    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"]
Oct 30 20:28:02.932034 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:28:02.932004    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ceeec8_d415_4c20_9afd_800fae5899e1.slice/crio-100ba7cf067a4a7888cf99b42564570e27e26aecc90a3c3c1c9507bcfe979559 WatchSource:0}: Error finding container 100ba7cf067a4a7888cf99b42564570e27e26aecc90a3c3c1c9507bcfe979559: Status 404 returned error can't find the container with id 100ba7cf067a4a7888cf99b42564570e27e26aecc90a3c3c1c9507bcfe979559
Oct 30 20:28:02.947949 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:02.947917    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks" event={"ID":"f1ceeec8-d415-4c20-9afd-800fae5899e1","Type":"ContainerStarted","Data":"100ba7cf067a4a7888cf99b42564570e27e26aecc90a3c3c1c9507bcfe979559"}
Oct 30 20:28:03.964119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:03.964081    4917 generic.go:356] "Generic (PLEG): container finished" podID="f1ceeec8-d415-4c20-9afd-800fae5899e1" containerID="86018f243cea93d51f16ad01147fcdff5c922f637fae43973817f9c5df057b8f" exitCode=0
Oct 30 20:28:03.964608 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:03.964168    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks" event={"ID":"f1ceeec8-d415-4c20-9afd-800fae5899e1","Type":"ContainerDied","Data":"86018f243cea93d51f16ad01147fcdff5c922f637fae43973817f9c5df057b8f"}
Oct 30 20:28:04.002142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:04.002109    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fd940b-caf5-410b-93e5-45759775b5bf" path="/var/lib/kubelet/pods/e3fd940b-caf5-410b-93e5-45759775b5bf/volumes"
Oct 30 20:28:05.111627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.111600    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:05.250310 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.250227    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca\") pod \"f1ceeec8-d415-4c20-9afd-800fae5899e1\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") "
Oct 30 20:28:05.250459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.250349    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45\" (UniqueName: \"kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45\") pod \"f1ceeec8-d415-4c20-9afd-800fae5899e1\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") "
Oct 30 20:28:05.250459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.250384    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca\") pod \"f1ceeec8-d415-4c20-9afd-800fae5899e1\" (UID: \"f1ceeec8-d415-4c20-9afd-800fae5899e1\") "
Oct 30 20:28:05.250840 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.250806    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca" pod "f1ceeec8-d415-4c20-9afd-800fae5899e1" (UID: "f1ceeec8-d415-4c20-9afd-800fae5899e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:05.252525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.252491    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca" pod "f1ceeec8-d415-4c20-9afd-800fae5899e1" (UID: "f1ceeec8-d415-4c20-9afd-800fae5899e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:05.253069 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.253043    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45" pod "f1ceeec8-d415-4c20-9afd-800fae5899e1" (UID: "f1ceeec8-d415-4c20-9afd-800fae5899e1"). InnerVolumeSpecName "kube-api-access-7hc45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:05.351519 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.351485    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:05.351519 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.351517    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc45\" (UniqueName: \"kubernetes.io/projected/f1ceeec8-d415-4c20-9afd-800fae5899e1-kube-api-access-7hc45\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:05.351519 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.351527    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ceeec8-d415-4c20-9afd-800fae5899e1-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:05.977252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.977223    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"
Oct 30 20:28:05.977252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.977235    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks" event={"ID":"f1ceeec8-d415-4c20-9afd-800fae5899e1","Type":"ContainerDied","Data":"100ba7cf067a4a7888cf99b42564570e27e26aecc90a3c3c1c9507bcfe979559"}
Oct 30 20:28:05.977514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:05.977280    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="100ba7cf067a4a7888cf99b42564570e27e26aecc90a3c3c1c9507bcfe979559"
Oct 30 20:28:06.894035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:06.894001    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"]
Oct 30 20:28:09.679348 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:28:09.679308    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:28:21.914156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:21.914108    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" containerID="cri-o://f7c63b058a446a6babaaa3b3c846eb3afd770b0421155e2b961dd195738f630c" gracePeriod=45
Oct 30 20:28:21.989782 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:21.989739    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 20:28:22.078680 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.078641    4917 generic.go:356] "Generic (PLEG): container finished" podID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerID="f7c63b058a446a6babaaa3b3c846eb3afd770b0421155e2b961dd195738f630c" exitCode=0
Oct 30 20:28:22.078864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.078686    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" event={"ID":"7372bcc3-6279-43ed-94b1-77889b4d9435","Type":"ContainerDied","Data":"f7c63b058a446a6babaaa3b3c846eb3afd770b0421155e2b961dd195738f630c"}
Oct 30 20:28:22.183651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.183625    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:28:22.221982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.221949    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir\") pod \"7372bcc3-6279-43ed-94b1-77889b4d9435\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") "
Oct 30 20:28:22.222143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.221992    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52\" (UniqueName: \"kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52\") pod \"7372bcc3-6279-43ed-94b1-77889b4d9435\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") "
Oct 30 20:28:22.222143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222027    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs\" (UniqueName: \"kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs\") pod \"7372bcc3-6279-43ed-94b1-77889b4d9435\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") "
Oct 30 20:28:22.222143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222076    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache\") pod \"7372bcc3-6279-43ed-94b1-77889b4d9435\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") "
Oct 30 20:28:22.222325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222234    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir" pod "7372bcc3-6279-43ed-94b1-77889b4d9435" (UID: "7372bcc3-6279-43ed-94b1-77889b4d9435"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.222325 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222248    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home\") pod \"7372bcc3-6279-43ed-94b1-77889b4d9435\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") "
Oct 30 20:28:22.222438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222359    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm\") pod \"7372bcc3-6279-43ed-94b1-77889b4d9435\" (UID: \"7372bcc3-6279-43ed-94b1-77889b4d9435\") "
Oct 30 20:28:22.222438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222377    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache" pod "7372bcc3-6279-43ed-94b1-77889b4d9435" (UID: "7372bcc3-6279-43ed-94b1-77889b4d9435"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.222532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222460    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home" pod "7372bcc3-6279-43ed-94b1-77889b4d9435" (UID: "7372bcc3-6279-43ed-94b1-77889b4d9435"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.222708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222686    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.222804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222712    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.222804 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.222722    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.224396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.224374    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52" pod "7372bcc3-6279-43ed-94b1-77889b4d9435" (UID: "7372bcc3-6279-43ed-94b1-77889b4d9435"). InnerVolumeSpecName "kube-api-access-7lp52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:22.224482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.224407    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs" pod "7372bcc3-6279-43ed-94b1-77889b4d9435" (UID: "7372bcc3-6279-43ed-94b1-77889b4d9435"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:22.224482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.224406    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm" pod "7372bcc3-6279-43ed-94b1-77889b4d9435" (UID: "7372bcc3-6279-43ed-94b1-77889b4d9435"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.323993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.323948    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7372bcc3-6279-43ed-94b1-77889b4d9435-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.323993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.323980    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lp52\" (UniqueName: \"kubernetes.io/projected/7372bcc3-6279-43ed-94b1-77889b4d9435-kube-api-access-7lp52\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.323993 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:22.323989    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7372bcc3-6279-43ed-94b1-77889b4d9435-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:23.086645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:23.086612    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"
Oct 30 20:28:23.087134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:23.086639    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52" event={"ID":"7372bcc3-6279-43ed-94b1-77889b4d9435","Type":"ContainerDied","Data":"b84bb257c893dde586da1b6e9eb139847167455d3a567b4a0ecfaa116c8ee9d3"}
Oct 30 20:28:23.087134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:23.086685    4917 scope.go:122] "RemoveContainer" containerID="f7c63b058a446a6babaaa3b3c846eb3afd770b0421155e2b961dd195738f630c"
Oct 30 20:28:23.253813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:23.253763    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"]
Oct 30 20:28:23.303474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:23.303441    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-w0-779921-kserve-9887bf5f6-v6z52"]
Oct 30 20:28:24.001640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:24.001597    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" path="/var/lib/kubelet/pods/7372bcc3-6279-43ed-94b1-77889b4d9435/volumes"
Oct 30 20:28:30.200376 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:30.200343    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:28:31.041563 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.041530    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"]
Oct 30 20:28:31.041965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.041951    4917 cpu_manager.go:414] "containerMap: removing container" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main"
Oct 30 20:28:31.042023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.041966    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main"
Oct 30 20:28:31.042023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.042006    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f1ceeec8-d415-4c20-9afd-800fae5899e1" containerName="revoke-keys"
Oct 30 20:28:31.042023 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.042012    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ceeec8-d415-4c20-9afd-800fae5899e1" containerName="revoke-keys"
Oct 30 20:28:31.042125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.042068    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="7372bcc3-6279-43ed-94b1-77889b4d9435" containerName="main"
Oct 30 20:28:31.042125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.042086    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ceeec8-d415-4c20-9afd-800fae5899e1" containerName="revoke-keys"
Oct 30 20:28:31.046623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.046605    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.100237 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.100206    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:31.108798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.108730    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.108798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.108787    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.109063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.109026    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.150495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.150465    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:31.150495 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.150483    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:31.150693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.150637    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:28:31.154981 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.154938    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"]
Oct 30 20:28:31.210551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.210517    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.211025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.210561    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.211025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.210659    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.211381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.211360    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.213796 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.213755    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.272718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.272687    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh\") pod \"maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.359349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.359279    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:31.734866 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:31.734834    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"]
Oct 30 20:28:31.735900 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:28:31.735871    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ad40f9_e0a0_49b5_b13d_994159079498.slice/crio-e83c086c43119ab814fe444230258f5c6b2a624cf03384d415eb1d7a4098d002 WatchSource:0}: Error finding container e83c086c43119ab814fe444230258f5c6b2a624cf03384d415eb1d7a4098d002: Status 404 returned error can't find the container with id e83c086c43119ab814fe444230258f5c6b2a624cf03384d415eb1d7a4098d002
Oct 30 20:28:32.056313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:32.056220    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"]
Oct 30 20:28:32.058610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:32.058586    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac56a5bd-77whx"]
Oct 30 20:28:32.146626 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:32.146591    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz" event={"ID":"31ad40f9-e0a0-49b5-b13d-994159079498","Type":"ContainerStarted","Data":"0cd2958d0011766899719e5e9bf87ac39df9e4fc1cd0d767682585c04e6d2622"}
Oct 30 20:28:32.146816 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:32.146645    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz" event={"ID":"31ad40f9-e0a0-49b5-b13d-994159079498","Type":"ContainerStarted","Data":"e83c086c43119ab814fe444230258f5c6b2a624cf03384d415eb1d7a4098d002"}
Oct 30 20:28:32.265756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:32.265704    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz" podStartSLOduration=2.265691059 podStartE2EDuration="2.265691059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:32.263485695 +0000 UTC m=+2808.427887334" watchObservedRunningTime="2026-10-30 20:28:32.265691059 +0000 UTC m=+2808.430092701"
Oct 30 20:28:33.154575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:33.154542    4917 generic.go:356] "Generic (PLEG): container finished" podID="31ad40f9-e0a0-49b5-b13d-994159079498" containerID="0cd2958d0011766899719e5e9bf87ac39df9e4fc1cd0d767682585c04e6d2622" exitCode=0
Oct 30 20:28:33.154798 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:33.154626    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz" event={"ID":"31ad40f9-e0a0-49b5-b13d-994159079498","Type":"ContainerDied","Data":"0cd2958d0011766899719e5e9bf87ac39df9e4fc1cd0d767682585c04e6d2622"}
Oct 30 20:28:34.002996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.002959    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b64805-fa8c-4882-b3de-882cc26ad733" path="/var/lib/kubelet/pods/18b64805-fa8c-4882-b3de-882cc26ad733/volumes"
Oct 30 20:28:34.306510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.306485    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:34.441102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.441071    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca\") pod \"31ad40f9-e0a0-49b5-b13d-994159079498\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") "
Oct 30 20:28:34.441272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.441147    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh\") pod \"31ad40f9-e0a0-49b5-b13d-994159079498\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") "
Oct 30 20:28:34.441272 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.441190    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca\") pod \"31ad40f9-e0a0-49b5-b13d-994159079498\" (UID: \"31ad40f9-e0a0-49b5-b13d-994159079498\") "
Oct 30 20:28:34.441576 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.441554    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca" pod "31ad40f9-e0a0-49b5-b13d-994159079498" (UID: "31ad40f9-e0a0-49b5-b13d-994159079498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:34.443401 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.443375    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca" pod "31ad40f9-e0a0-49b5-b13d-994159079498" (UID: "31ad40f9-e0a0-49b5-b13d-994159079498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:34.443514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.443457    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh" pod "31ad40f9-e0a0-49b5-b13d-994159079498" (UID: "31ad40f9-e0a0-49b5-b13d-994159079498"). InnerVolumeSpecName "kube-api-access-855rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:34.542083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.542006    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:34.542083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.542034    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/31ad40f9-e0a0-49b5-b13d-994159079498-kube-api-access-855rh\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:34.542083 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:34.542044    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ad40f9-e0a0-49b5-b13d-994159079498-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:35.168694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:35.168659    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz" event={"ID":"31ad40f9-e0a0-49b5-b13d-994159079498","Type":"ContainerDied","Data":"e83c086c43119ab814fe444230258f5c6b2a624cf03384d415eb1d7a4098d002"}
Oct 30 20:28:35.169154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:35.168709    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83c086c43119ab814fe444230258f5c6b2a624cf03384d415eb1d7a4098d002"
Oct 30 20:28:35.169154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:35.168676    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"
Oct 30 20:28:39.679420 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:28:39.679383    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:28:42.285475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.285435    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"]
Oct 30 20:28:42.285925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.285908    4917 cpu_manager.go:414] "containerMap: removing container" podUID="31ad40f9-e0a0-49b5-b13d-994159079498" containerName="revoke-keys"
Oct 30 20:28:42.285979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.285926    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ad40f9-e0a0-49b5-b13d-994159079498" containerName="revoke-keys"
Oct 30 20:28:42.286040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.286030    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ad40f9-e0a0-49b5-b13d-994159079498" containerName="revoke-keys"
Oct 30 20:28:42.289291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.289272    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.342537 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.342504    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:42.342824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.342541    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:42.342824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.342689    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:42.342824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.342720    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:28:42.397080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.397050    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"]
Oct 30 20:28:42.420917 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.420880    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.421080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.420939    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnfc\" (UniqueName: \"kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.421224 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.421186    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.521911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.521879    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.522076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.521921    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnfc\" (UniqueName: \"kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.522076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.522014    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.522737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.522713    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.524379 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.524357    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.583921 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.583857    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnfc\" (UniqueName: \"kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc\") pod \"maas-api-revoke-sub-ab9a460d-j5jcm\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.603683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.603661    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:42.988332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:42.988290    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"]
Oct 30 20:28:42.990479 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:28:42.990453    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f07688_df10_48c8_bb20_b9756a4a6d9e.slice/crio-288aa312ca9299c03026bb4a60390f16f4fe186c8fd5c73368ee01ddcf16e817 WatchSource:0}: Error finding container 288aa312ca9299c03026bb4a60390f16f4fe186c8fd5c73368ee01ddcf16e817: Status 404 returned error can't find the container with id 288aa312ca9299c03026bb4a60390f16f4fe186c8fd5c73368ee01ddcf16e817
Oct 30 20:28:43.221107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:43.221066    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm" event={"ID":"66f07688-df10-48c8-bb20-b9756a4a6d9e","Type":"ContainerStarted","Data":"288aa312ca9299c03026bb4a60390f16f4fe186c8fd5c73368ee01ddcf16e817"}
Oct 30 20:28:44.230435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:44.230404    4917 generic.go:356] "Generic (PLEG): container finished" podID="66f07688-df10-48c8-bb20-b9756a4a6d9e" containerID="43c5c1082a7b7d9b51c50660b75b9ea7b059ed2c81d45c0c4291b37215bdfd01" exitCode=0
Oct 30 20:28:44.230961 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:44.230489    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm" event={"ID":"66f07688-df10-48c8-bb20-b9756a4a6d9e","Type":"ContainerDied","Data":"43c5c1082a7b7d9b51c50660b75b9ea7b059ed2c81d45c0c4291b37215bdfd01"}
Oct 30 20:28:45.382095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.382070    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:45.453362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.453328    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca\") pod \"66f07688-df10-48c8-bb20-b9756a4a6d9e\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") "
Oct 30 20:28:45.453547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.453433    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc\" (UniqueName: \"kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc\") pod \"66f07688-df10-48c8-bb20-b9756a4a6d9e\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") "
Oct 30 20:28:45.453547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.453529    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca\") pod \"66f07688-df10-48c8-bb20-b9756a4a6d9e\" (UID: \"66f07688-df10-48c8-bb20-b9756a4a6d9e\") "
Oct 30 20:28:45.453739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.453714    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca" pod "66f07688-df10-48c8-bb20-b9756a4a6d9e" (UID: "66f07688-df10-48c8-bb20-b9756a4a6d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:45.453994 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.453967    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:45.455656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.455633    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc" pod "66f07688-df10-48c8-bb20-b9756a4a6d9e" (UID: "66f07688-df10-48c8-bb20-b9756a4a6d9e"). InnerVolumeSpecName "kube-api-access-nrnfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:45.455753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.455635    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca" pod "66f07688-df10-48c8-bb20-b9756a4a6d9e" (UID: "66f07688-df10-48c8-bb20-b9756a4a6d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:45.555294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.555203    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrnfc\" (UniqueName: \"kubernetes.io/projected/66f07688-df10-48c8-bb20-b9756a4a6d9e-kube-api-access-nrnfc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:45.555294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.555236    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f07688-df10-48c8-bb20-b9756a4a6d9e-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:45.916719 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:45.916642    4917 scope.go:122] "RemoveContainer" containerID="52bb9297763e5b72d48d86fd496cc81ba7d4b70892ed7ec479a23d0d22ac29bc"
Oct 30 20:28:46.253148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:46.253118    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"
Oct 30 20:28:46.253327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:46.253118    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm" event={"ID":"66f07688-df10-48c8-bb20-b9756a4a6d9e","Type":"ContainerDied","Data":"288aa312ca9299c03026bb4a60390f16f4fe186c8fd5c73368ee01ddcf16e817"}
Oct 30 20:28:46.253327 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:46.253268    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288aa312ca9299c03026bb4a60390f16f4fe186c8fd5c73368ee01ddcf16e817"
Oct 30 20:28:47.020967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.020930    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"]
Oct 30 20:28:47.021551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.021530    4917 cpu_manager.go:414] "containerMap: removing container" podUID="66f07688-df10-48c8-bb20-b9756a4a6d9e" containerName="revoke-keys"
Oct 30 20:28:47.021643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.021555    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f07688-df10-48c8-bb20-b9756a4a6d9e" containerName="revoke-keys"
Oct 30 20:28:47.021703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.021688    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f07688-df10-48c8-bb20-b9756a4a6d9e" containerName="revoke-keys"
Oct 30 20:28:47.026307 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.026285    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.082541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.082497    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:47.084831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.084801    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"]
Oct 30 20:28:47.130414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.130379    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:47.130598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.130497    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:47.130651 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.130628    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:28:47.170936 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.170903    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.171136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.171044    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwf8\" (UniqueName: \"kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.171221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.171164    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.272140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.272048    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwf8\" (UniqueName: \"kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.272140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.272133    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.272374 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.272196    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.272803 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.272761    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.274530 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.274510    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.335841 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.335811    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwf8\" (UniqueName: \"kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8\") pod \"maas-api-revoke-sub-b6c700f5-nnjlw\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.340535 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.340506    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:47.720612 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:47.720580    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"]
Oct 30 20:28:47.721910 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:28:47.721879    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddeb808_ae9b_4b13_b985_a4268a27492f.slice/crio-4b621147a4e340bf4d58ae8e8bef2e0aee3fe2793a370666b18e7832e78404b1 WatchSource:0}: Error finding container 4b621147a4e340bf4d58ae8e8bef2e0aee3fe2793a370666b18e7832e78404b1: Status 404 returned error can't find the container with id 4b621147a4e340bf4d58ae8e8bef2e0aee3fe2793a370666b18e7832e78404b1
Oct 30 20:28:48.266154 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:48.266117    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw" event={"ID":"0ddeb808-ae9b-4b13-b985-a4268a27492f","Type":"ContainerStarted","Data":"a1a412b9f40fd76624a157e0fbf3df128c45bce4d34b9b8606497f576804f1de"}
Oct 30 20:28:48.266663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:48.266185    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw" event={"ID":"0ddeb808-ae9b-4b13-b985-a4268a27492f","Type":"ContainerStarted","Data":"4b621147a4e340bf4d58ae8e8bef2e0aee3fe2793a370666b18e7832e78404b1"}
Oct 30 20:28:48.386547 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:48.386444    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw" podStartSLOduration=2.386430716 podStartE2EDuration="2.386430716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:48.3835823 +0000 UTC m=+2824.547983946" watchObservedRunningTime="2026-10-30 20:28:48.386430716 +0000 UTC m=+2824.550832358"
Oct 30 20:28:49.273590 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:49.273554    4917 generic.go:356] "Generic (PLEG): container finished" podID="0ddeb808-ae9b-4b13-b985-a4268a27492f" containerID="a1a412b9f40fd76624a157e0fbf3df128c45bce4d34b9b8606497f576804f1de" exitCode=0
Oct 30 20:28:49.274022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:49.273597    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw" event={"ID":"0ddeb808-ae9b-4b13-b985-a4268a27492f","Type":"ContainerDied","Data":"a1a412b9f40fd76624a157e0fbf3df128c45bce4d34b9b8606497f576804f1de"}
Oct 30 20:28:49.490339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:49.490304    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"]
Oct 30 20:28:49.494351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:49.494319    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-01e8dc-s5mmz"]
Oct 30 20:28:50.001643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.001609    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ad40f9-e0a0-49b5-b13d-994159079498" path="/var/lib/kubelet/pods/31ad40f9-e0a0-49b5-b13d-994159079498/volumes"
Oct 30 20:28:50.425544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.425522    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:50.606979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.606900    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca\") pod \"0ddeb808-ae9b-4b13-b985-a4268a27492f\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") "
Oct 30 20:28:50.607134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.606981    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8\" (UniqueName: \"kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8\") pod \"0ddeb808-ae9b-4b13-b985-a4268a27492f\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") "
Oct 30 20:28:50.607134 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.607072    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca\") pod \"0ddeb808-ae9b-4b13-b985-a4268a27492f\" (UID: \"0ddeb808-ae9b-4b13-b985-a4268a27492f\") "
Oct 30 20:28:50.607344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.607324    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca" pod "0ddeb808-ae9b-4b13-b985-a4268a27492f" (UID: "0ddeb808-ae9b-4b13-b985-a4268a27492f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:50.607428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.607413    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:50.609158 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.609132    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca" pod "0ddeb808-ae9b-4b13-b985-a4268a27492f" (UID: "0ddeb808-ae9b-4b13-b985-a4268a27492f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:50.609262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.609188    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8" pod "0ddeb808-ae9b-4b13-b985-a4268a27492f" (UID: "0ddeb808-ae9b-4b13-b985-a4268a27492f"). InnerVolumeSpecName "kube-api-access-qzwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:50.708258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.708226    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwf8\" (UniqueName: \"kubernetes.io/projected/0ddeb808-ae9b-4b13-b985-a4268a27492f-kube-api-access-qzwf8\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:50.708258 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:50.708254    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddeb808-ae9b-4b13-b985-a4268a27492f-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:51.288787 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:51.288742    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"
Oct 30 20:28:51.288978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:51.288794    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw" event={"ID":"0ddeb808-ae9b-4b13-b985-a4268a27492f","Type":"ContainerDied","Data":"4b621147a4e340bf4d58ae8e8bef2e0aee3fe2793a370666b18e7832e78404b1"}
Oct 30 20:28:51.288978 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:51.288870    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b621147a4e340bf4d58ae8e8bef2e0aee3fe2793a370666b18e7832e78404b1"
Oct 30 20:28:54.370145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:54.370072    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:28:59.252765 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.252725    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"]
Oct 30 20:28:59.253184 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.253171    4917 cpu_manager.go:414] "containerMap: removing container" podUID="0ddeb808-ae9b-4b13-b985-a4268a27492f" containerName="revoke-keys"
Oct 30 20:28:59.253228 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.253186    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddeb808-ae9b-4b13-b985-a4268a27492f" containerName="revoke-keys"
Oct 30 20:28:59.253263 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.253254    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddeb808-ae9b-4b13-b985-a4268a27492f" containerName="revoke-keys"
Oct 30 20:28:59.256532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.256516    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.288398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.288372    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.288542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.288483    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.288629 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.288608    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qdv\" (UniqueName: \"kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.314700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.314671    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"]
Oct 30 20:28:59.362059 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.362027    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:28:59.362312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.362075    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:59.362312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.362130    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:28:59.362312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.362155    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:28:59.389235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.389203    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qdv\" (UniqueName: \"kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.389375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.389246    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.389375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.389289    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.389979 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.389952    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.391728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.391707    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.463932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.463898    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qdv\" (UniqueName: \"kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv\") pod \"maas-api-revoke-keys-e2e-inf-b-453277-9dpxl\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.569431 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.569349    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:28:59.958427 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:28:59.958398    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"]
Oct 30 20:28:59.959367 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:28:59.959335    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7e9ff9_3657_4887_beb0_b2ae77293783.slice/crio-f8d723db0bf2863c58ff917c69c5ccfbd026d90cdfc333aa96ff495748b9d6a4 WatchSource:0}: Error finding container f8d723db0bf2863c58ff917c69c5ccfbd026d90cdfc333aa96ff495748b9d6a4: Status 404 returned error can't find the container with id f8d723db0bf2863c58ff917c69c5ccfbd026d90cdfc333aa96ff495748b9d6a4
Oct 30 20:29:00.346855 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:00.346810    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl" event={"ID":"3d7e9ff9-3657-4887-beb0-b2ae77293783","Type":"ContainerStarted","Data":"86ffcd5040424fea30386492af9a4fdc5f6cb5739ad5fb6a101d6e163397d602"}
Oct 30 20:29:00.347363 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:00.346879    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl" event={"ID":"3d7e9ff9-3657-4887-beb0-b2ae77293783","Type":"ContainerStarted","Data":"f8d723db0bf2863c58ff917c69c5ccfbd026d90cdfc333aa96ff495748b9d6a4"}
Oct 30 20:29:00.466485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:00.466429    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl" podStartSLOduration=1.466415072 podStartE2EDuration="1.466415072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:00.463926022 +0000 UTC m=+2836.628327678" watchObservedRunningTime="2026-10-30 20:29:00.466415072 +0000 UTC m=+2836.630816710"
Oct 30 20:29:01.355617 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:01.355581    4917 generic.go:356] "Generic (PLEG): container finished" podID="3d7e9ff9-3657-4887-beb0-b2ae77293783" containerID="86ffcd5040424fea30386492af9a4fdc5f6cb5739ad5fb6a101d6e163397d602" exitCode=0
Oct 30 20:29:01.356168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:01.355659    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl" event={"ID":"3d7e9ff9-3657-4887-beb0-b2ae77293783","Type":"ContainerDied","Data":"86ffcd5040424fea30386492af9a4fdc5f6cb5739ad5fb6a101d6e163397d602"}
Oct 30 20:29:02.499528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.499499    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:29:02.514250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.514165    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv\" (UniqueName: \"kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv\") pod \"3d7e9ff9-3657-4887-beb0-b2ae77293783\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") "
Oct 30 20:29:02.514396 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.514307    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca\") pod \"3d7e9ff9-3657-4887-beb0-b2ae77293783\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") "
Oct 30 20:29:02.514470 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.514420    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca\") pod \"3d7e9ff9-3657-4887-beb0-b2ae77293783\" (UID: \"3d7e9ff9-3657-4887-beb0-b2ae77293783\") "
Oct 30 20:29:02.514832 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.514800    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca" pod "3d7e9ff9-3657-4887-beb0-b2ae77293783" (UID: "3d7e9ff9-3657-4887-beb0-b2ae77293783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:02.516399 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.516373    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv" pod "3d7e9ff9-3657-4887-beb0-b2ae77293783" (UID: "3d7e9ff9-3657-4887-beb0-b2ae77293783"). InnerVolumeSpecName "kube-api-access-q7qdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:02.516487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.516413    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca" pod "3d7e9ff9-3657-4887-beb0-b2ae77293783" (UID: "3d7e9ff9-3657-4887-beb0-b2ae77293783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:02.615666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.615575    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7qdv\" (UniqueName: \"kubernetes.io/projected/3d7e9ff9-3657-4887-beb0-b2ae77293783-kube-api-access-q7qdv\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:02.615666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.615606    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:02.615666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:02.615621    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e9ff9-3657-4887-beb0-b2ae77293783-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:03.368830 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:03.368794    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl" event={"ID":"3d7e9ff9-3657-4887-beb0-b2ae77293783","Type":"ContainerDied","Data":"f8d723db0bf2863c58ff917c69c5ccfbd026d90cdfc333aa96ff495748b9d6a4"}
Oct 30 20:29:03.368990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:03.368831    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"
Oct 30 20:29:03.368990 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:03.368841    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d723db0bf2863c58ff917c69c5ccfbd026d90cdfc333aa96ff495748b9d6a4"
Oct 30 20:29:08.051377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:08.051341    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"]
Oct 30 20:29:08.101569 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:08.101537    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221df18b-gn27p"]
Oct 30 20:29:09.057142 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:09.057106    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"]
Oct 30 20:29:09.107631 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:09.107599    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c9af9e-sqjnd"]
Oct 30 20:29:09.680230 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:29:09.680189    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:29:10.001457 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:10.001377    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9" path="/var/lib/kubelet/pods/7bc2c3cc-d6e7-47e0-bd10-d038ac00f6f9/volumes"
Oct 30 20:29:10.001911 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:10.001894    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6bb20b0-c159-4221-a734-103a05b35eff" path="/var/lib/kubelet/pods/d6bb20b0-c159-4221-a734-103a05b35eff/volumes"
Oct 30 20:29:12.920157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:12.920122    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:29:13.338602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.338564    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"]
Oct 30 20:29:13.339001 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.338987    4917 cpu_manager.go:414] "containerMap: removing container" podUID="3d7e9ff9-3657-4887-beb0-b2ae77293783" containerName="revoke-keys"
Oct 30 20:29:13.339056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.339001    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7e9ff9-3657-4887-beb0-b2ae77293783" containerName="revoke-keys"
Oct 30 20:29:13.339099 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.339085    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7e9ff9-3657-4887-beb0-b2ae77293783" containerName="revoke-keys"
Oct 30 20:29:13.342288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.342272    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.397945 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.397920    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:13.398191 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.397949    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:29:13.398191 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.397944    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:29:13.418613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.418572    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.418787 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.418638    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.418919 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.418896    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btvk\" (UniqueName: \"kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.448408 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.448324    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:29:13.455828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.455795    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"]
Oct 30 20:29:13.519844 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.519811    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.520027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.519969    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btvk\" (UniqueName: \"kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.520071 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.520033    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.520694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.520669    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.522428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.522403    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.582145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.582113    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btvk\" (UniqueName: \"kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk\") pod \"maas-api-revoke-sub-2cef4a9c-j4n9v\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:13.656647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:13.656561    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:14.035440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:14.035407    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"]
Oct 30 20:29:14.036831 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:29:14.036800    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf352e693_eead_442d_92db_0068374d8216.slice/crio-fad727ba1a1336703afa3a8ce177943ee2db1e7f72de51c13064e68819ad225e WatchSource:0}: Error finding container fad727ba1a1336703afa3a8ce177943ee2db1e7f72de51c13064e68819ad225e: Status 404 returned error can't find the container with id fad727ba1a1336703afa3a8ce177943ee2db1e7f72de51c13064e68819ad225e
Oct 30 20:29:14.456996 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:14.456952    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v" event={"ID":"f352e693-eead-442d-92db-0068374d8216","Type":"ContainerStarted","Data":"2f1d5424b7d657bf3d95dd76a775664f7045f73ca4e96b5424204d964aaaf6b4"}
Oct 30 20:29:14.457220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:14.457006    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v" event={"ID":"f352e693-eead-442d-92db-0068374d8216","Type":"ContainerStarted","Data":"fad727ba1a1336703afa3a8ce177943ee2db1e7f72de51c13064e68819ad225e"}
Oct 30 20:29:14.582929 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:14.582873    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v" podStartSLOduration=1.582859445 podStartE2EDuration="1.582859445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:14.579623274 +0000 UTC m=+2850.744024916" watchObservedRunningTime="2026-10-30 20:29:14.582859445 +0000 UTC m=+2850.747261134"
Oct 30 20:29:15.465027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:15.464994    4917 generic.go:356] "Generic (PLEG): container finished" podID="f352e693-eead-442d-92db-0068374d8216" containerID="2f1d5424b7d657bf3d95dd76a775664f7045f73ca4e96b5424204d964aaaf6b4" exitCode=0
Oct 30 20:29:15.465383 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:15.465077    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v" event={"ID":"f352e693-eead-442d-92db-0068374d8216","Type":"ContainerDied","Data":"2f1d5424b7d657bf3d95dd76a775664f7045f73ca4e96b5424204d964aaaf6b4"}
Oct 30 20:29:16.612133 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.612105    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:16.652697 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.652670    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca\") pod \"f352e693-eead-442d-92db-0068374d8216\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") "
Oct 30 20:29:16.652887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.652815    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca\") pod \"f352e693-eead-442d-92db-0068374d8216\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") "
Oct 30 20:29:16.652887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.652858    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk\" (UniqueName: \"kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk\") pod \"f352e693-eead-442d-92db-0068374d8216\" (UID: \"f352e693-eead-442d-92db-0068374d8216\") "
Oct 30 20:29:16.653220 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.653192    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca" pod "f352e693-eead-442d-92db-0068374d8216" (UID: "f352e693-eead-442d-92db-0068374d8216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:16.655019 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.654997    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca" pod "f352e693-eead-442d-92db-0068374d8216" (UID: "f352e693-eead-442d-92db-0068374d8216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:16.655430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.655410    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk" pod "f352e693-eead-442d-92db-0068374d8216" (UID: "f352e693-eead-442d-92db-0068374d8216"). InnerVolumeSpecName "kube-api-access-4btvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:16.754818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.754717    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f352e693-eead-442d-92db-0068374d8216-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:16.754818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.754746    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f352e693-eead-442d-92db-0068374d8216-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:16.754818 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:16.754760    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4btvk\" (UniqueName: \"kubernetes.io/projected/f352e693-eead-442d-92db-0068374d8216-kube-api-access-4btvk\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:17.483514 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:17.483481    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"
Oct 30 20:29:17.483737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:17.483478    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v" event={"ID":"f352e693-eead-442d-92db-0068374d8216","Type":"ContainerDied","Data":"fad727ba1a1336703afa3a8ce177943ee2db1e7f72de51c13064e68819ad225e"}
Oct 30 20:29:17.483737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:17.483608    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad727ba1a1336703afa3a8ce177943ee2db1e7f72de51c13064e68819ad225e"
Oct 30 20:29:20.774524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:20.774488    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"]
Oct 30 20:29:20.825065 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:20.825032    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779921-z9dks"]
Oct 30 20:29:22.002312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:22.002278    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ceeec8-d415-4c20-9afd-800fae5899e1" path="/var/lib/kubelet/pods/f1ceeec8-d415-4c20-9afd-800fae5899e1/volumes"
Oct 30 20:29:27.081745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:27.081708    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"]
Oct 30 20:29:27.132111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:27.132077    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453277-9dpxl"]
Oct 30 20:29:28.001789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:28.001749    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7e9ff9-3657-4887-beb0-b2ae77293783" path="/var/lib/kubelet/pods/3d7e9ff9-3657-4887-beb0-b2ae77293783/volumes"
Oct 30 20:29:28.356587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:28.353556    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"]
Oct 30 20:29:28.356587 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:28.353945    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy" containerID="cri-o://0851fdb4dcba61ab9950df67ae211331e5324c985e8ef16fed37bf6ae09e763d" gracePeriod=30
Oct 30 20:29:33.583243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.583204    4917 generic.go:356] "Generic (PLEG): container finished" podID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerID="0851fdb4dcba61ab9950df67ae211331e5324c985e8ef16fed37bf6ae09e763d" exitCode=0
Oct 30 20:29:33.583642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.583279    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" event={"ID":"eadfae5c-c853-456d-bd09-e6f904b8a4b7","Type":"ContainerDied","Data":"0851fdb4dcba61ab9950df67ae211331e5324c985e8ef16fed37bf6ae09e763d"}
Oct 30 20:29:33.612789 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.612755    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:29:33.723427 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723396    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.723620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723464    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.723620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723493    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.723620 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723591    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.723819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723627    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.723819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723706    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:33.723819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723794    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.724063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723838    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:33.724063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723853    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.724063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723888    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.724063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723914    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy\") pod \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\" (UID: \"eadfae5c-c853-456d-bd09-e6f904b8a4b7\") "
Oct 30 20:29:33.724063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.723930    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:33.724300 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724115    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:33.724300 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724046    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:33.724373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724327    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.724373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724347    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.724373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724362    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.724503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724377    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.724503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.724390    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.725944 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.725907    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:33.726063 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.726030    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:33.726200 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.726181    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:29:33.726265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.726200    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq" pod "eadfae5c-c853-456d-bd09-e6f904b8a4b7" (UID: "eadfae5c-c853-456d-bd09-e6f904b8a4b7"). InnerVolumeSpecName "kube-api-access-c6jlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:33.825302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.825271    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.825302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.825298    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6jlq\" (UniqueName: \"kubernetes.io/projected/eadfae5c-c853-456d-bd09-e6f904b8a4b7-kube-api-access-c6jlq\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.825302 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.825310    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:33.825529 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:33.825319    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadfae5c-c853-456d-bd09-e6f904b8a4b7-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:34.591188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:34.591158    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"
Oct 30 20:29:34.591633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:34.591182    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc" event={"ID":"eadfae5c-c853-456d-bd09-e6f904b8a4b7","Type":"ContainerDied","Data":"e5cc984b07265c0055c07e591ca1348e4e3140f765eb427e8e8c334a45dcdbb7"}
Oct 30 20:29:34.591633 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:34.591276    4917 scope.go:122] "RemoveContainer" containerID="0851fdb4dcba61ab9950df67ae211331e5324c985e8ef16fed37bf6ae09e763d"
Oct 30 20:29:34.761868 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:34.761826    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"]
Oct 30 20:29:34.764723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:34.764698    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-d4nkc"]
Oct 30 20:29:36.001883 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:36.001848    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" path="/var/lib/kubelet/pods/eadfae5c-c853-456d-bd09-e6f904b8a4b7/volumes"
Oct 30 20:29:39.679381 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:29:39.679347    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:29:40.419179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:40.419140    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"]
Oct 30 20:29:40.421674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:40.421630    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"]
Oct 30 20:29:40.424090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:40.424062    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"]
Oct 30 20:29:40.501579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:40.501544    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:29:45.961670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:45.961635    4917 scope.go:122] "RemoveContainer" containerID="97b1f4a610560d9546abc2c9a17885aff120294dc691c26e429ab9934909e14b"
Oct 30 20:29:45.971088 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:45.971067    4917 scope.go:122] "RemoveContainer" containerID="f7311183398d8a860ffc146aeae44ba5ade7102fcca9c9f639aa47ebedc3d77c"
Oct 30 20:29:45.979971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:45.979932    4917 scope.go:122] "RemoveContainer" containerID="b477bd2ee9489faf8b9c864c7e65532ed057d20d06d4951156843bef3f6b7c20"
Oct 30 20:29:47.247976 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:47.247939    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"]
Oct 30 20:29:55.440428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.440380    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main" containerID="cri-o://a5b44da9faedcaf6dd80273ea0320d39246b4b986fc99b0216ba8dd3128167de" gracePeriod=45
Oct 30 20:29:55.441395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.441361    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" containerID="cri-o://51997c9cbb94742803c3f1f5b2c506f05615201554710b541e221b7e4e377a83" gracePeriod=45
Oct 30 20:29:55.443011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.442977    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main" containerID="cri-o://1bf00595c6ee1950569ef24cda6b6667c2d57917dc32afa0be62b9490dc9a5fe" gracePeriod=45
Oct 30 20:29:55.650273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.650230    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": dial tcp 10.132.0.54:8000: connect: connection refused"
Oct 30 20:29:55.722454 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.722425    4917 generic.go:356] "Generic (PLEG): container finished" podID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerID="1bf00595c6ee1950569ef24cda6b6667c2d57917dc32afa0be62b9490dc9a5fe" exitCode=0
Oct 30 20:29:55.722674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.722636    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" event={"ID":"17d2bbfa-e4ff-4489-a673-61d5aac0dadf","Type":"ContainerDied","Data":"1bf00595c6ee1950569ef24cda6b6667c2d57917dc32afa0be62b9490dc9a5fe"}
Oct 30 20:29:55.722758 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.722699    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp" event={"ID":"17d2bbfa-e4ff-4489-a673-61d5aac0dadf","Type":"ContainerDied","Data":"cb4eab443f970c437aaf6f37526e8e399f804046496f2bee4e0755ae25bdb662"}
Oct 30 20:29:55.722758 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.722718    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4eab443f970c437aaf6f37526e8e399f804046496f2bee4e0755ae25bdb662"
Oct 30 20:29:55.724755 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.724724    4917 generic.go:356] "Generic (PLEG): container finished" podID="704961c4-062f-4508-9c50-d37652ea729d" containerID="a5b44da9faedcaf6dd80273ea0320d39246b4b986fc99b0216ba8dd3128167de" exitCode=0
Oct 30 20:29:55.724926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.724841    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" event={"ID":"704961c4-062f-4508-9c50-d37652ea729d","Type":"ContainerDied","Data":"a5b44da9faedcaf6dd80273ea0320d39246b4b986fc99b0216ba8dd3128167de"}
Oct 30 20:29:55.726736 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.726714    4917 generic.go:356] "Generic (PLEG): container finished" podID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerID="51997c9cbb94742803c3f1f5b2c506f05615201554710b541e221b7e4e377a83" exitCode=0
Oct 30 20:29:55.726885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.726789    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" event={"ID":"e02c50a5-c8a4-4382-a23c-a051e3dc23c4","Type":"ContainerDied","Data":"51997c9cbb94742803c3f1f5b2c506f05615201554710b541e221b7e4e377a83"}
Oct 30 20:29:55.729602 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.729582    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:29:55.761876 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.761845    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:29:55.764893 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.764868    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:29:55.840572 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840534    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home\") pod \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") "
Oct 30 20:29:55.840793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840582    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home\") pod \"704961c4-062f-4508-9c50-d37652ea729d\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") "
Oct 30 20:29:55.840793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840618    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs\") pod \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") "
Oct 30 20:29:55.840793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840684    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache\") pod \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") "
Oct 30 20:29:55.840793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840721    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw\" (UniqueName: \"kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw\") pod \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") "
Oct 30 20:29:55.840793 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840753    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs\") pod \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") "
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840810    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home\") pod \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") "
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840849    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache\") pod \"704961c4-062f-4508-9c50-d37652ea729d\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") "
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840876    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm\") pod \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") "
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840908    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home" pod "704961c4-062f-4508-9c50-d37652ea729d" (UID: "704961c4-062f-4508-9c50-d37652ea729d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840916    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home" pod "17d2bbfa-e4ff-4489-a673-61d5aac0dadf" (UID: "17d2bbfa-e4ff-4489-a673-61d5aac0dadf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840914    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm\") pod \"704961c4-062f-4508-9c50-d37652ea729d\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") "
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.840993    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache" pod "17d2bbfa-e4ff-4489-a673-61d5aac0dadf" (UID: "17d2bbfa-e4ff-4489-a673-61d5aac0dadf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841003    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz\" (UniqueName: \"kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz\") pod \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") "
Oct 30 20:29:55.841098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841069    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir\") pod \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841122    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs\" (UniqueName: \"kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs\") pod \"704961c4-062f-4508-9c50-d37652ea729d\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841161    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir\") pod \"704961c4-062f-4508-9c50-d37652ea729d\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841199    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir\") pod \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841243    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm\") pod \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\" (UID: \"17d2bbfa-e4ff-4489-a673-61d5aac0dadf\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841272    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5\" (UniqueName: \"kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5\") pod \"704961c4-062f-4508-9c50-d37652ea729d\" (UID: \"704961c4-062f-4508-9c50-d37652ea729d\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841306    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache\") pod \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\" (UID: \"e02c50a5-c8a4-4382-a23c-a051e3dc23c4\") "
Oct 30 20:29:55.841577 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841382    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache" pod "704961c4-062f-4508-9c50-d37652ea729d" (UID: "704961c4-062f-4508-9c50-d37652ea729d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841588    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home" pod "e02c50a5-c8a4-4382-a23c-a051e3dc23c4" (UID: "e02c50a5-c8a4-4382-a23c-a051e3dc23c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841812    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache" pod "e02c50a5-c8a4-4382-a23c-a051e3dc23c4" (UID: "e02c50a5-c8a4-4382-a23c-a051e3dc23c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841888    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841908    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841924    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841938    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.841988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841950    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.842443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.841998    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir" pod "e02c50a5-c8a4-4382-a23c-a051e3dc23c4" (UID: "e02c50a5-c8a4-4382-a23c-a051e3dc23c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.842443 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.842136    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir" pod "17d2bbfa-e4ff-4489-a673-61d5aac0dadf" (UID: "17d2bbfa-e4ff-4489-a673-61d5aac0dadf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.842966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.842873    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir" pod "704961c4-062f-4508-9c50-d37652ea729d" (UID: "704961c4-062f-4508-9c50-d37652ea729d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.844628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.844597    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm" pod "e02c50a5-c8a4-4382-a23c-a051e3dc23c4" (UID: "e02c50a5-c8a4-4382-a23c-a051e3dc23c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.844748 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.844685    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs" pod "17d2bbfa-e4ff-4489-a673-61d5aac0dadf" (UID: "17d2bbfa-e4ff-4489-a673-61d5aac0dadf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:55.844971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.844942    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz" pod "17d2bbfa-e4ff-4489-a673-61d5aac0dadf" (UID: "17d2bbfa-e4ff-4489-a673-61d5aac0dadf"). InnerVolumeSpecName "kube-api-access-2vnfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:55.844971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.844953    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm" pod "704961c4-062f-4508-9c50-d37652ea729d" (UID: "704961c4-062f-4508-9c50-d37652ea729d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.845162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.844966    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5" pod "704961c4-062f-4508-9c50-d37652ea729d" (UID: "704961c4-062f-4508-9c50-d37652ea729d"). InnerVolumeSpecName "kube-api-access-zmtj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:55.845162 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.844978    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs" pod "e02c50a5-c8a4-4382-a23c-a051e3dc23c4" (UID: "e02c50a5-c8a4-4382-a23c-a051e3dc23c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:55.845301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.845280    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm" pod "17d2bbfa-e4ff-4489-a673-61d5aac0dadf" (UID: "17d2bbfa-e4ff-4489-a673-61d5aac0dadf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:29:55.846177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.846154    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs" pod "704961c4-062f-4508-9c50-d37652ea729d" (UID: "704961c4-062f-4508-9c50-d37652ea729d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:55.846317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.846297    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw" pod "e02c50a5-c8a4-4382-a23c-a051e3dc23c4" (UID: "e02c50a5-c8a4-4382-a23c-a051e3dc23c4"). InnerVolumeSpecName "kube-api-access-rs6mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:55.943375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943338    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnfz\" (UniqueName: \"kubernetes.io/projected/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-kube-api-access-2vnfz\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943369    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943379    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704961c4-062f-4508-9c50-d37652ea729d-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943388    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943397    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943404    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943413    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmtj5\" (UniqueName: \"kubernetes.io/projected/704961c4-062f-4508-9c50-d37652ea729d-kube-api-access-zmtj5\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943423    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943432    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d2bbfa-e4ff-4489-a673-61d5aac0dadf-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943440    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6mw\" (UniqueName: \"kubernetes.io/projected/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-kube-api-access-rs6mw\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943447    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943455    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c50a5-c8a4-4382-a23c-a051e3dc23c4-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:55.943622 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:55.943462    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704961c4-062f-4508-9c50-d37652ea729d-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:56.733968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.733873    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"
Oct 30 20:29:56.733968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.733899    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9" event={"ID":"e02c50a5-c8a4-4382-a23c-a051e3dc23c4","Type":"ContainerDied","Data":"464db2870a771f51fdd6d46f0830c3d7cbb3bafb2431c8d89c1a71231d107723"}
Oct 30 20:29:56.733968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.733963    4917 scope.go:122] "RemoveContainer" containerID="51997c9cbb94742803c3f1f5b2c506f05615201554710b541e221b7e4e377a83"
Oct 30 20:29:56.735792 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.735739    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb" event={"ID":"704961c4-062f-4508-9c50-d37652ea729d","Type":"ContainerDied","Data":"768195a573dd68b1cee9b467f0c98c4198e4751121d480779a012f674426a604"}
Oct 30 20:29:56.735927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.735804    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"
Oct 30 20:29:56.735927 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.735819    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"
Oct 30 20:29:56.745522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.745502    4917 scope.go:122] "RemoveContainer" containerID="a5b44da9faedcaf6dd80273ea0320d39246b4b986fc99b0216ba8dd3128167de"
Oct 30 20:29:56.901754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.901716    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"]
Oct 30 20:29:56.951865 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:56.951829    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-407632/simulator-w5-407632-kserve-696559477b-ckrgp"]
Oct 30 20:29:57.016221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:57.016139    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"]
Oct 30 20:29:57.018544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:57.018519    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-407632/distinct-2-w5-407632-kserve-658fdbbdf9-h7vmb"]
Oct 30 20:29:57.132498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:57.132463    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"]
Oct 30 20:29:57.137701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:57.137673    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-407632/premium-w5-407632-kserve-5d797b6cd5-fgfl9"]
Oct 30 20:29:58.001647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:58.001613    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" path="/var/lib/kubelet/pods/17d2bbfa-e4ff-4489-a673-61d5aac0dadf/volumes"
Oct 30 20:29:58.002078 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:58.002063    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704961c4-062f-4508-9c50-d37652ea729d" path="/var/lib/kubelet/pods/704961c4-062f-4508-9c50-d37652ea729d/volumes"
Oct 30 20:29:58.002441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:29:58.002428    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" path="/var/lib/kubelet/pods/e02c50a5-c8a4-4382-a23c-a051e3dc23c4/volumes"
Oct 30 20:30:00.170235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170201    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"]
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170599    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170611    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170620    4917 cpu_manager.go:414] "containerMap: removing container" podUID="f352e693-eead-442d-92db-0068374d8216" containerName="revoke-keys"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170627    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f352e693-eead-442d-92db-0068374d8216" containerName="revoke-keys"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170652    4917 cpu_manager.go:414] "containerMap: removing container" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170657    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170663    4917 cpu_manager.go:414] "containerMap: removing container" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main"
Oct 30 20:30:00.170674 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170668    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170685    4917 cpu_manager.go:414] "containerMap: removing container" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170690    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170761    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02c50a5-c8a4-4382-a23c-a051e3dc23c4" containerName="main"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170786    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="f352e693-eead-442d-92db-0068374d8216" containerName="revoke-keys"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170799    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="eadfae5c-c853-456d-bd09-e6f904b8a4b7" containerName="istio-proxy"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170808    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="704961c4-062f-4508-9c50-d37652ea729d" containerName="main"
Oct 30 20:30:00.171000 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.170819    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d2bbfa-e4ff-4489-a673-61d5aac0dadf" containerName="main"
Oct 30 20:30:00.175192 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.175172    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.177276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.177252    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"]
Oct 30 20:30:00.180754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.180735    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.228298 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.228259    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:30:00.228511 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.228492    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:00.228603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.228583    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:00.272599 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.272563    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"]
Oct 30 20:30:00.282206 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.282173    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"]
Oct 30 20:30:00.287349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.287313    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dlzg\" (UniqueName: \"kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.287522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.287454    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.287588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.287523    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmvr\" (UniqueName: \"kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr\") pod \"maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.287588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.287560    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.388068 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.388026    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dlzg\" (UniqueName: \"kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.388282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.388099    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.388282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.388134    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmvr\" (UniqueName: \"kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr\") pod \"maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.388282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.388161    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.388871 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.388849    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.388926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.388857    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.450657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.450624    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dlzg\" (UniqueName: \"kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.450848 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.450624    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmvr\" (UniqueName: \"kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr\") pod \"maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.494862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.494835    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:00.504541 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.504515    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:00.879831 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.879802    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"]
Oct 30 20:30:00.881061 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:30:00.881035    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b18eed_71c6_49ea_87f9_561fe91dcd22.slice/crio-5182a85b4851c60f78f4a3ee7bcc58b220f7fed03c670eedb6999c5e7bedae0d WatchSource:0}: Error finding container 5182a85b4851c60f78f4a3ee7bcc58b220f7fed03c670eedb6999c5e7bedae0d: Status 404 returned error can't find the container with id 5182a85b4851c60f78f4a3ee7bcc58b220f7fed03c670eedb6999c5e7bedae0d
Oct 30 20:30:00.929825 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:00.929798    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"]
Oct 30 20:30:00.931188 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:30:00.931150    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60addf8c_add3_4bad_bd45_6596ce7cf5ce.slice/crio-cdc38be24105b46e1e0164c9a3c00be79f1b461e74e06371bcd9893580483ef4 WatchSource:0}: Error finding container cdc38be24105b46e1e0164c9a3c00be79f1b461e74e06371bcd9893580483ef4: Status 404 returned error can't find the container with id cdc38be24105b46e1e0164c9a3c00be79f1b461e74e06371bcd9893580483ef4
Oct 30 20:30:01.218163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.218127    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"]
Oct 30 20:30:01.221924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.221904    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.279810 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.279751    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:01.280049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.279999    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"]
Oct 30 20:30:01.297227 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.297186    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjf4m\" (UniqueName: \"kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.297403 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.297256    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.297403 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.297334    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.397974 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.397939    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjf4m\" (UniqueName: \"kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.398155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.398041    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.398155 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.398088    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.398856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.398833    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.400526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.400506    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.482098 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.482060    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjf4m\" (UniqueName: \"kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m\") pod \"maas-api-revoke-sub-b8de3fef-wktbf\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.535013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.534973    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:01.772980 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.772942    4917 generic.go:356] "Generic (PLEG): container finished" podID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerID="9aa557e35b106aaa736e762823103e530eebf516f85b8ea3a072971e3c921d22" exitCode=60
Oct 30 20:30:01.773171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.773032    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" event={"ID":"60addf8c-add3-4bad-bd45-6596ce7cf5ce","Type":"ContainerDied","Data":"9aa557e35b106aaa736e762823103e530eebf516f85b8ea3a072971e3c921d22"}
Oct 30 20:30:01.773171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.773088    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" event={"ID":"60addf8c-add3-4bad-bd45-6596ce7cf5ce","Type":"ContainerStarted","Data":"cdc38be24105b46e1e0164c9a3c00be79f1b461e74e06371bcd9893580483ef4"}
Oct 30 20:30:01.773332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.773313    4917 scope.go:122] "RemoveContainer" containerID="9aa557e35b106aaa736e762823103e530eebf516f85b8ea3a072971e3c921d22"
Oct 30 20:30:01.774829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.774731    4917 generic.go:356] "Generic (PLEG): container finished" podID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerID="3724c72369376e20d3385e92ec9b0d012f5638a38943f527b2c8c67e375c653c" exitCode=60
Oct 30 20:30:01.774829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.774799    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" event={"ID":"e5b18eed-71c6-49ea-87f9-561fe91dcd22","Type":"ContainerDied","Data":"3724c72369376e20d3385e92ec9b0d012f5638a38943f527b2c8c67e375c653c"}
Oct 30 20:30:01.774937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.774848    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" event={"ID":"e5b18eed-71c6-49ea-87f9-561fe91dcd22","Type":"ContainerStarted","Data":"5182a85b4851c60f78f4a3ee7bcc58b220f7fed03c670eedb6999c5e7bedae0d"}
Oct 30 20:30:01.775061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.775046    4917 scope.go:122] "RemoveContainer" containerID="3724c72369376e20d3385e92ec9b0d012f5638a38943f527b2c8c67e375c653c"
Oct 30 20:30:01.911463 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:01.911434    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"]
Oct 30 20:30:01.912903 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:30:01.912877    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ef6971_4b73_4d6b_bfba_1f75681fe4fb.slice/crio-468c8062fbdcbdcbca0d5702cca6d9bccb6d63c4cb691a4f38057ec649446699 WatchSource:0}: Error finding container 468c8062fbdcbdcbca0d5702cca6d9bccb6d63c4cb691a4f38057ec649446699: Status 404 returned error can't find the container with id 468c8062fbdcbdcbca0d5702cca6d9bccb6d63c4cb691a4f38057ec649446699
Oct 30 20:30:02.267717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.267677    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv" podUID="40593e05-8672-4df6-9e84-a64da221f298" containerName="main" containerID="cri-o://e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239" gracePeriod=45
Oct 30 20:30:02.537072 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.537046    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:30:02.609721 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609686    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home\") pod \"40593e05-8672-4df6-9e84-a64da221f298\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") "
Oct 30 20:30:02.609924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609735    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache\") pod \"40593e05-8672-4df6-9e84-a64da221f298\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") "
Oct 30 20:30:02.609924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609789    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir\") pod \"40593e05-8672-4df6-9e84-a64da221f298\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") "
Oct 30 20:30:02.609924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609865    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm\") pod \"40593e05-8672-4df6-9e84-a64da221f298\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") "
Oct 30 20:30:02.610058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609931    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx\" (UniqueName: \"kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx\") pod \"40593e05-8672-4df6-9e84-a64da221f298\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") "
Oct 30 20:30:02.610058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609987    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs\" (UniqueName: \"kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs\") pod \"40593e05-8672-4df6-9e84-a64da221f298\" (UID: \"40593e05-8672-4df6-9e84-a64da221f298\") "
Oct 30 20:30:02.610058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.609999    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home" pod "40593e05-8672-4df6-9e84-a64da221f298" (UID: "40593e05-8672-4df6-9e84-a64da221f298"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:02.610058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.610016    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache" pod "40593e05-8672-4df6-9e84-a64da221f298" (UID: "40593e05-8672-4df6-9e84-a64da221f298"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:02.610058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.610035    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir" pod "40593e05-8672-4df6-9e84-a64da221f298" (UID: "40593e05-8672-4df6-9e84-a64da221f298"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:02.610295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.610280    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:02.610331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.610297    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:02.610331 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.610310    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:02.612156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.612132    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm" pod "40593e05-8672-4df6-9e84-a64da221f298" (UID: "40593e05-8672-4df6-9e84-a64da221f298"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:02.612294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.612216    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs" pod "40593e05-8672-4df6-9e84-a64da221f298" (UID: "40593e05-8672-4df6-9e84-a64da221f298"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:02.612362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.612342    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx" pod "40593e05-8672-4df6-9e84-a64da221f298" (UID: "40593e05-8672-4df6-9e84-a64da221f298"). InnerVolumeSpecName "kube-api-access-wstbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:02.711402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.711367    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40593e05-8672-4df6-9e84-a64da221f298-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:02.711402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.711399    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstbx\" (UniqueName: \"kubernetes.io/projected/40593e05-8672-4df6-9e84-a64da221f298-kube-api-access-wstbx\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:02.711402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.711409    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40593e05-8672-4df6-9e84-a64da221f298-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:02.782641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.782605    4917 generic.go:356] "Generic (PLEG): container finished" podID="40593e05-8672-4df6-9e84-a64da221f298" containerID="e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239" exitCode=0
Oct 30 20:30:02.782846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.782653    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"
Oct 30 20:30:02.782846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.782687    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv" event={"ID":"40593e05-8672-4df6-9e84-a64da221f298","Type":"ContainerDied","Data":"e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239"}
Oct 30 20:30:02.782846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.782732    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv" event={"ID":"40593e05-8672-4df6-9e84-a64da221f298","Type":"ContainerDied","Data":"4fc15a40c69273261aca2aec0cdea4b706e3b50e7bd04937fd883dd3a6f00589"}
Oct 30 20:30:02.782846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.782754    4917 scope.go:122] "RemoveContainer" containerID="e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239"
Oct 30 20:30:02.784932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.784906    4917 generic.go:356] "Generic (PLEG): container finished" podID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerID="af2cdd51aa4cbd2b9bbb683aad73b99e01da03f3b54c62c35fdb65eeb1366185" exitCode=60
Oct 30 20:30:02.785040 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.784981    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" event={"ID":"60addf8c-add3-4bad-bd45-6596ce7cf5ce","Type":"ContainerDied","Data":"af2cdd51aa4cbd2b9bbb683aad73b99e01da03f3b54c62c35fdb65eeb1366185"}
Oct 30 20:30:02.785324 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.785305    4917 scope.go:122] "RemoveContainer" containerID="af2cdd51aa4cbd2b9bbb683aad73b99e01da03f3b54c62c35fdb65eeb1366185"
Oct 30 20:30:02.785667 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:02.785644    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj_odh-ai-gateway-infra(60addf8c-add3-4bad-bd45-6596ce7cf5ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce"
Oct 30 20:30:02.786817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.786765    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf" event={"ID":"01ef6971-4b73-4d6b-bfba-1f75681fe4fb","Type":"ContainerStarted","Data":"95c46c03515c56f81d0cb5476bc8538ae1908d0e2075624f61c912c8906c4252"}
Oct 30 20:30:02.786940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.786841    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf" event={"ID":"01ef6971-4b73-4d6b-bfba-1f75681fe4fb","Type":"ContainerStarted","Data":"468c8062fbdcbdcbca0d5702cca6d9bccb6d63c4cb691a4f38057ec649446699"}
Oct 30 20:30:02.788758 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.788739    4917 generic.go:356] "Generic (PLEG): container finished" podID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerID="656716da5bb4f2a1b5a55460dbd3fc8d18e332c2f6fb5b9322a00796e75e0a65" exitCode=60
Oct 30 20:30:02.788856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.788792    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" event={"ID":"e5b18eed-71c6-49ea-87f9-561fe91dcd22","Type":"ContainerDied","Data":"656716da5bb4f2a1b5a55460dbd3fc8d18e332c2f6fb5b9322a00796e75e0a65"}
Oct 30 20:30:02.789038 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.789025    4917 scope.go:122] "RemoveContainer" containerID="656716da5bb4f2a1b5a55460dbd3fc8d18e332c2f6fb5b9322a00796e75e0a65"
Oct 30 20:30:02.789240 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:02.789224    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl_odh-ai-gateway-infra(e5b18eed-71c6-49ea-87f9-561fe91dcd22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22"
Oct 30 20:30:02.857956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.857930    4917 scope.go:122] "RemoveContainer" containerID="e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239"
Oct 30 20:30:02.858294 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:02.858275    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239\": container with ID starting with e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239 not found: ID does not exist" containerID="e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239"
Oct 30 20:30:02.858339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.858307    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239"} err="failed to get container status \"e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239\": rpc error: code = NotFound desc = could not find container \"e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239\": container with ID starting with e65f596443760588bef5624c985a83c3c21f87115fbc1ea482188f2eeba6f239 not found: ID does not exist"
Oct 30 20:30:02.858339 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.858325    4917 scope.go:122] "RemoveContainer" containerID="9aa557e35b106aaa736e762823103e530eebf516f85b8ea3a072971e3c921d22"
Oct 30 20:30:02.871493 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:02.871471    4917 scope.go:122] "RemoveContainer" containerID="3724c72369376e20d3385e92ec9b0d012f5638a38943f527b2c8c67e375c653c"
Oct 30 20:30:03.007846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.007748    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:30:03.077143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.077080    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf" podStartSLOduration=2.077063252 podStartE2EDuration="2.077063252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.076552014 +0000 UTC m=+2899.240953659" watchObservedRunningTime="2026-10-30 20:30:03.077063252 +0000 UTC m=+2899.241464897"
Oct 30 20:30:03.317650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.317558    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"]
Oct 30 20:30:03.323279 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.323244    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-72db69/test-model-72db69-kserve-6c4df7c859-wcgkv"]
Oct 30 20:30:03.798564 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.798535    4917 scope.go:122] "RemoveContainer" containerID="af2cdd51aa4cbd2b9bbb683aad73b99e01da03f3b54c62c35fdb65eeb1366185"
Oct 30 20:30:03.798871 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:03.798847    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj_odh-ai-gateway-infra(60addf8c-add3-4bad-bd45-6596ce7cf5ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce"
Oct 30 20:30:03.799829 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.799804    4917 generic.go:356] "Generic (PLEG): container finished" podID="01ef6971-4b73-4d6b-bfba-1f75681fe4fb" containerID="95c46c03515c56f81d0cb5476bc8538ae1908d0e2075624f61c912c8906c4252" exitCode=0
Oct 30 20:30:03.799983 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.799889    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf" event={"ID":"01ef6971-4b73-4d6b-bfba-1f75681fe4fb","Type":"ContainerDied","Data":"95c46c03515c56f81d0cb5476bc8538ae1908d0e2075624f61c912c8906c4252"}
Oct 30 20:30:03.801627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:03.801610    4917 scope.go:122] "RemoveContainer" containerID="656716da5bb4f2a1b5a55460dbd3fc8d18e332c2f6fb5b9322a00796e75e0a65"
Oct 30 20:30:03.801855 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:03.801836    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl_odh-ai-gateway-infra(e5b18eed-71c6-49ea-87f9-561fe91dcd22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22"
Oct 30 20:30:04.001173 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:04.001132    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40593e05-8672-4df6-9e84-a64da221f298" path="/var/lib/kubelet/pods/40593e05-8672-4df6-9e84-a64da221f298/volumes"
Oct 30 20:30:04.955691 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:04.955663    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:05.034908 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.034875    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca\") pod \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") "
Oct 30 20:30:05.035073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.034957    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m\" (UniqueName: \"kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m\") pod \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") "
Oct 30 20:30:05.035073 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.035035    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca\") pod \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\" (UID: \"01ef6971-4b73-4d6b-bfba-1f75681fe4fb\") "
Oct 30 20:30:05.035373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.035339    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca" pod "01ef6971-4b73-4d6b-bfba-1f75681fe4fb" (UID: "01ef6971-4b73-4d6b-bfba-1f75681fe4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:05.037191 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.037165    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m" pod "01ef6971-4b73-4d6b-bfba-1f75681fe4fb" (UID: "01ef6971-4b73-4d6b-bfba-1f75681fe4fb"). InnerVolumeSpecName "kube-api-access-kjf4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:05.037267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.037172    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca" pod "01ef6971-4b73-4d6b-bfba-1f75681fe4fb" (UID: "01ef6971-4b73-4d6b-bfba-1f75681fe4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:05.136395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.136307    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:05.136395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.136341    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjf4m\" (UniqueName: \"kubernetes.io/projected/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-kube-api-access-kjf4m\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:05.136395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.136352    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ef6971-4b73-4d6b-bfba-1f75681fe4fb-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:05.815897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.815864    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"
Oct 30 20:30:05.816090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.815911    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf" event={"ID":"01ef6971-4b73-4d6b-bfba-1f75681fe4fb","Type":"ContainerDied","Data":"468c8062fbdcbdcbca0d5702cca6d9bccb6d63c4cb691a4f38057ec649446699"}
Oct 30 20:30:05.816090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:05.815961    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468c8062fbdcbdcbca0d5702cca6d9bccb6d63c4cb691a4f38057ec649446699"
Oct 30 20:30:09.679471 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:09.679427    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:30:13.998728 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:13.998693    4917 scope.go:122] "RemoveContainer" containerID="656716da5bb4f2a1b5a55460dbd3fc8d18e332c2f6fb5b9322a00796e75e0a65"
Oct 30 20:30:13.999202 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:13.998802    4917 scope.go:122] "RemoveContainer" containerID="af2cdd51aa4cbd2b9bbb683aad73b99e01da03f3b54c62c35fdb65eeb1366185"
Oct 30 20:30:14.873106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:14.873019    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" event={"ID":"e5b18eed-71c6-49ea-87f9-561fe91dcd22","Type":"ContainerStarted","Data":"9e5c9e22c201bca75ad12463c54fc2b40aa86cc6a23ca8582465ddf28e33634f"}
Oct 30 20:30:14.875160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:14.875131    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" event={"ID":"60addf8c-add3-4bad-bd45-6596ce7cf5ce","Type":"ContainerStarted","Data":"51ad935bb6cb108f1437e96744c35b07ecaaba6784027fa00dd7f4bd4c191197"}
Oct 30 20:30:14.992251 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:14.992195    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" podStartSLOduration=14.992181584 podStartE2EDuration="14.992181584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:14.990036803 +0000 UTC m=+2911.154438445" watchObservedRunningTime="2026-10-30 20:30:14.992181584 +0000 UTC m=+2911.156583226"
Oct 30 20:30:15.112784 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.112713    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" podStartSLOduration=15.112698672 podStartE2EDuration="15.112698672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:15.109859822 +0000 UTC m=+2911.274261489" watchObservedRunningTime="2026-10-30 20:30:15.112698672 +0000 UTC m=+2911.277100323"
Oct 30 20:30:15.884211 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.884174    4917 generic.go:356] "Generic (PLEG): container finished" podID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerID="9e5c9e22c201bca75ad12463c54fc2b40aa86cc6a23ca8582465ddf28e33634f" exitCode=60
Oct 30 20:30:15.884438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.884247    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" event={"ID":"e5b18eed-71c6-49ea-87f9-561fe91dcd22","Type":"ContainerDied","Data":"9e5c9e22c201bca75ad12463c54fc2b40aa86cc6a23ca8582465ddf28e33634f"}
Oct 30 20:30:15.884438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.884331    4917 scope.go:122] "RemoveContainer" containerID="656716da5bb4f2a1b5a55460dbd3fc8d18e332c2f6fb5b9322a00796e75e0a65"
Oct 30 20:30:15.884711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.884687    4917 scope.go:122] "RemoveContainer" containerID="9e5c9e22c201bca75ad12463c54fc2b40aa86cc6a23ca8582465ddf28e33634f"
Oct 30 20:30:15.885018 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:15.884988    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl_odh-ai-gateway-infra(e5b18eed-71c6-49ea-87f9-561fe91dcd22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22"
Oct 30 20:30:15.886424 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.886405    4917 generic.go:356] "Generic (PLEG): container finished" podID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerID="51ad935bb6cb108f1437e96744c35b07ecaaba6784027fa00dd7f4bd4c191197" exitCode=60
Oct 30 20:30:15.886515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.886473    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" event={"ID":"60addf8c-add3-4bad-bd45-6596ce7cf5ce","Type":"ContainerDied","Data":"51ad935bb6cb108f1437e96744c35b07ecaaba6784027fa00dd7f4bd4c191197"}
Oct 30 20:30:15.886807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.886793    4917 scope.go:122] "RemoveContainer" containerID="51ad935bb6cb108f1437e96744c35b07ecaaba6784027fa00dd7f4bd4c191197"
Oct 30 20:30:15.887014 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:15.886998    4917 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj_odh-ai-gateway-infra(60addf8c-add3-4bad-bd45-6596ce7cf5ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce"
Oct 30 20:30:15.896896 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:15.896872    4917 scope.go:122] "RemoveContainer" containerID="af2cdd51aa4cbd2b9bbb683aad73b99e01da03f3b54c62c35fdb65eeb1366185"
Oct 30 20:30:16.021972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:16.021934    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"]
Oct 30 20:30:16.128288 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:16.128247    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"]
Oct 30 20:30:17.030837 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.030805    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:30:17.039640 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.039616    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:17.078814 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.078765    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:17.160539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.160451    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr\" (UniqueName: \"kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr\") pod \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") "
Oct 30 20:30:17.160539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.160513    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca\") pod \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\" (UID: \"e5b18eed-71c6-49ea-87f9-561fe91dcd22\") "
Oct 30 20:30:17.161035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.160602    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca\") pod \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") "
Oct 30 20:30:17.161035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.160632    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg\" (UniqueName: \"kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg\") pod \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\" (UID: \"60addf8c-add3-4bad-bd45-6596ce7cf5ce\") "
Oct 30 20:30:17.161035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.160928    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca" pod "e5b18eed-71c6-49ea-87f9-561fe91dcd22" (UID: "e5b18eed-71c6-49ea-87f9-561fe91dcd22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:17.161035 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.161030    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca" pod "60addf8c-add3-4bad-bd45-6596ce7cf5ce" (UID: "60addf8c-add3-4bad-bd45-6596ce7cf5ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:17.161283 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.161054    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b18eed-71c6-49ea-87f9-561fe91dcd22-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.162997 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.162974    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg" pod "60addf8c-add3-4bad-bd45-6596ce7cf5ce" (UID: "60addf8c-add3-4bad-bd45-6596ce7cf5ce"). InnerVolumeSpecName "kube-api-access-2dlzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:17.163219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.163197    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr" pod "e5b18eed-71c6-49ea-87f9-561fe91dcd22" (UID: "e5b18eed-71c6-49ea-87f9-561fe91dcd22"). InnerVolumeSpecName "kube-api-access-jnmvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:17.262322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.262288    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60addf8c-add3-4bad-bd45-6596ce7cf5ce-maas-api-service-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.262322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.262318    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dlzg\" (UniqueName: \"kubernetes.io/projected/60addf8c-add3-4bad-bd45-6596ce7cf5ce-kube-api-access-2dlzg\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.262322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.262328    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmvr\" (UniqueName: \"kubernetes.io/projected/e5b18eed-71c6-49ea-87f9-561fe91dcd22-kube-api-access-jnmvr\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.904869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.904835    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"
Oct 30 20:30:17.905102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.904888    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj" event={"ID":"60addf8c-add3-4bad-bd45-6596ce7cf5ce","Type":"ContainerDied","Data":"cdc38be24105b46e1e0164c9a3c00be79f1b461e74e06371bcd9893580483ef4"}
Oct 30 20:30:17.905102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.904989    4917 scope.go:122] "RemoveContainer" containerID="51ad935bb6cb108f1437e96744c35b07ecaaba6784027fa00dd7f4bd4c191197"
Oct 30 20:30:17.906503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.906471    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"
Oct 30 20:30:17.906503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.906467    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl" event={"ID":"e5b18eed-71c6-49ea-87f9-561fe91dcd22","Type":"ContainerDied","Data":"5182a85b4851c60f78f4a3ee7bcc58b220f7fed03c670eedb6999c5e7bedae0d"}
Oct 30 20:30:17.916971 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:17.916945    4917 scope.go:122] "RemoveContainer" containerID="9e5c9e22c201bca75ad12463c54fc2b40aa86cc6a23ca8582465ddf28e33634f"
Oct 30 20:30:18.076414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:18.076378    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"]
Oct 30 20:30:18.084510 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:18.084479    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889870-97kzj"]
Oct 30 20:30:18.204895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:18.204859    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"]
Oct 30 20:30:18.212070 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:18.212037    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-14c846-29889870-nb9vl"]
Oct 30 20:30:20.000925 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:20.000893    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" path="/var/lib/kubelet/pods/60addf8c-add3-4bad-bd45-6596ce7cf5ce/volumes"
Oct 30 20:30:20.001317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:20.001303    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" path="/var/lib/kubelet/pods/e5b18eed-71c6-49ea-87f9-561fe91dcd22/volumes"
Oct 30 20:30:31.123278 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:31.123188    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"]
Oct 30 20:30:31.123747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:31.123411    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy" containerID="cri-o://4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770" gracePeriod=30
Oct 30 20:30:36.376670 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.376640    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 20:30:36.454526 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454494    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.454722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454544    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.454722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454578    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.454722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454615    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.454722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454640    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.454722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454673    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.454722 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454711    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.455105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454814    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.455105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454869    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket\") pod \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\" (UID: \"377bbb6e-4fb5-47b4-be12-995d2dd14f89\") "
Oct 30 20:30:36.455105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.454901    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:36.455105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455010    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:36.455105 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455032    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:36.455318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455115    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:36.455318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455156    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:36.455318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455291    4917 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.455484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455335    4917 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-data\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.455484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455350    4917 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istiod-ca-cert\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.455484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455365    4917 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-workload-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.455484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.455380    4917 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-credential-socket\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.456932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.456900    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:36.457111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.457091    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "kube-api-access-cpllc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:36.457167 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.457141    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:36.457246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.457226    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token" pod "377bbb6e-4fb5-47b4-be12-995d2dd14f89" (UID: "377bbb6e-4fb5-47b4-be12-995d2dd14f89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:36.556759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.556725    4917 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-token\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.556759 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.556758    4917 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-envoy\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.557020 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.556796    4917 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bbb6e-4fb5-47b4-be12-995d2dd14f89-istio-podinfo\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:36.557020 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:36.556810    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpllc\" (UniqueName: \"kubernetes.io/projected/377bbb6e-4fb5-47b4-be12-995d2dd14f89-kube-api-access-cpllc\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:37.023436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.023398    4917 generic.go:356] "Generic (PLEG): container finished" podID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerID="4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770" exitCode=0
Oct 30 20:30:37.023662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.023457    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"
Oct 30 20:30:37.023662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.023494    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" event={"ID":"377bbb6e-4fb5-47b4-be12-995d2dd14f89","Type":"ContainerDied","Data":"4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770"}
Oct 30 20:30:37.023662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.023567    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n" event={"ID":"377bbb6e-4fb5-47b4-be12-995d2dd14f89","Type":"ContainerDied","Data":"bcd104197b097c31e95d2c30891eaf4cb4f9ed9d6d639e7cf8e946bbbf335fc9"}
Oct 30 20:30:37.023662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.023596    4917 scope.go:122] "RemoveContainer" containerID="4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770"
Oct 30 20:30:37.034025 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.034006    4917 scope.go:122] "RemoveContainer" containerID="2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"
Oct 30 20:30:37.042949 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.042929    4917 scope.go:122] "RemoveContainer" containerID="4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770"
Oct 30 20:30:37.043214 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:37.043194    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770\": container with ID starting with 4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770 not found: ID does not exist" containerID="4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770"
Oct 30 20:30:37.043267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.043225    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770"} err="failed to get container status \"4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770\": rpc error: code = NotFound desc = could not find container \"4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770\": container with ID starting with 4068e690f364411abf9ff042bcd242f52befae6a1eb8593402fae2a22c033770 not found: ID does not exist"
Oct 30 20:30:37.043267 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.043246    4917 scope.go:122] "RemoveContainer" containerID="2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"
Oct 30 20:30:37.043479 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:37.043464    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92\": container with ID starting with 2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92 not found: ID does not exist" containerID="2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"
Oct 30 20:30:37.043521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.043484    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92"} err="failed to get container status \"2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92\": rpc error: code = NotFound desc = could not find container \"2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92\": container with ID starting with 2db55e0556bedf4e38a829ea7404f0044dbc697dedd35a86cffdeb7b11fe9b92 not found: ID does not exist"
Oct 30 20:30:37.191643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.191609    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"]
Oct 30 20:30:37.241701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:37.241658    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d49n"]
Oct 30 20:30:38.001649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:38.001616    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" path="/var/lib/kubelet/pods/377bbb6e-4fb5-47b4-be12-995d2dd14f89/volumes"
Oct 30 20:30:39.686036 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:30:39.685997    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:30:43.815250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:43.815211    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:30:46.022241 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:46.022206    4917 scope.go:122] "RemoveContainer" containerID="c4a2a950c2a9259f2541259cf7a51b078115d6644602ac8ddddc3dc9bf3ba673"
Oct 30 20:30:46.031404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:46.031380    4917 scope.go:122] "RemoveContainer" containerID="1bf00595c6ee1950569ef24cda6b6667c2d57917dc32afa0be62b9490dc9a5fe"
Oct 30 20:30:46.040307 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:46.040287    4917 scope.go:122] "RemoveContainer" containerID="a4a22237ecfaf3363fe85caaf8a34abed0fd10788d93a7e3c0608f5fb89af512"
Oct 30 20:30:50.053924 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:50.053886    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"]
Oct 30 20:30:50.104897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:50.104861    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234dbf7-bbfc7"]
Oct 30 20:30:52.001213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:52.001180    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef098152-3a2b-437d-8633-dbe02247f55f" path="/var/lib/kubelet/pods/ef098152-3a2b-437d-8633-dbe02247f55f/volumes"
Oct 30 20:30:55.006186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:30:55.006148    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:31:03.734092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:03.734050    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"]
Oct 30 20:31:03.785399 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:03.785356    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"]
Oct 30 20:31:03.787794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:03.787747    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"]
Oct 30 20:31:09.679205 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:31:09.679165    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:31:09.821111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821065    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"]
Oct 30 20:31:09.821540 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821521    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821543    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821557    4917 cpu_manager.go:414] "containerMap: removing container" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821565    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821582    4917 cpu_manager.go:414] "containerMap: removing container" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821590    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821615    4917 cpu_manager.go:414] "containerMap: removing container" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821622    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821634    4917 cpu_manager.go:414] "containerMap: removing container" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.821643 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821641    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821658    4917 cpu_manager.go:414] "containerMap: removing container" podUID="40593e05-8672-4df6-9e84-a64da221f298" containerName="main"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821668    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40593e05-8672-4df6-9e84-a64da221f298" containerName="main"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821688    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821696    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821708    4917 cpu_manager.go:414] "containerMap: removing container" podUID="01ef6971-4b73-4d6b-bfba-1f75681fe4fb" containerName="revoke-keys"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821717    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ef6971-4b73-4d6b-bfba-1f75681fe4fb" containerName="revoke-keys"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821842    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821855    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821864    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821880    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821889    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821898    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821908    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="40593e05-8672-4df6-9e84-a64da221f298" containerName="main"
Oct 30 20:31:09.822161 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.821919    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ef6971-4b73-4d6b-bfba-1f75681fe4fb" containerName="revoke-keys"
Oct 30 20:31:09.827136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.827113    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.879735 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.879700    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.879948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.879765    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwv9\" (UniqueName: \"kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.879948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.879878    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.880948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.880924    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:09.881058 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.880951    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:31:09.881168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.881155    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:31:09.931711 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.931620    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:31:09.934751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.934726    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"]
Oct 30 20:31:09.981362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.981324    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.981574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.981394    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwv9\" (UniqueName: \"kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.981574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.981512    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.982053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.982032    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:09.983931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:09.983911    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:10.044562 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:10.044533    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwv9\" (UniqueName: \"kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9\") pod \"maas-api-revoke-sub-ff41f2ef-dflqt\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:10.140370 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:10.140333    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:10.514174 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:10.514134    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"]
Oct 30 20:31:10.515214 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:31:10.515188    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509e10d8_e99f_4500_af37_409428048394.slice/crio-fa3e1cc6ea1e58fd454fba8b90f1759e54352cff7098ee006e6fa6b4b02a5104 WatchSource:0}: Error finding container fa3e1cc6ea1e58fd454fba8b90f1759e54352cff7098ee006e6fa6b4b02a5104: Status 404 returned error can't find the container with id fa3e1cc6ea1e58fd454fba8b90f1759e54352cff7098ee006e6fa6b4b02a5104
Oct 30 20:31:10.875387 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:10.875350    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"]
Oct 30 20:31:11.223677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:11.223626    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt" event={"ID":"509e10d8-e99f-4500-af37-409428048394","Type":"ContainerStarted","Data":"f67df3be59c26e1aef6479129653dd2c643048b0f6debe72dcd046cad2d2d08a"}
Oct 30 20:31:11.223677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:11.223684    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt" event={"ID":"509e10d8-e99f-4500-af37-409428048394","Type":"ContainerStarted","Data":"fa3e1cc6ea1e58fd454fba8b90f1759e54352cff7098ee006e6fa6b4b02a5104"}
Oct 30 20:31:11.343956 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:11.343903    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt" podStartSLOduration=2.34388641 podStartE2EDuration="2.34388641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:11.341514371 +0000 UTC m=+2967.505916015" watchObservedRunningTime="2026-10-30 20:31:11.34388641 +0000 UTC m=+2967.508288051"
Oct 30 20:31:12.230425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:12.230389    4917 generic.go:356] "Generic (PLEG): container finished" podID="509e10d8-e99f-4500-af37-409428048394" containerID="f67df3be59c26e1aef6479129653dd2c643048b0f6debe72dcd046cad2d2d08a" exitCode=0
Oct 30 20:31:12.230889 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:12.230467    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt" event={"ID":"509e10d8-e99f-4500-af37-409428048394","Type":"ContainerDied","Data":"f67df3be59c26e1aef6479129653dd2c643048b0f6debe72dcd046cad2d2d08a"}
Oct 30 20:31:13.375834 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.375801    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:13.413008 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.412977    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca\") pod \"509e10d8-e99f-4500-af37-409428048394\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") "
Oct 30 20:31:13.413169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.413036    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9\" (UniqueName: \"kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9\") pod \"509e10d8-e99f-4500-af37-409428048394\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") "
Oct 30 20:31:13.413169 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.413111    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca\") pod \"509e10d8-e99f-4500-af37-409428048394\" (UID: \"509e10d8-e99f-4500-af37-409428048394\") "
Oct 30 20:31:13.413521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.413492    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca" pod "509e10d8-e99f-4500-af37-409428048394" (UID: "509e10d8-e99f-4500-af37-409428048394"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:13.415301 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.415275    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca" pod "509e10d8-e99f-4500-af37-409428048394" (UID: "509e10d8-e99f-4500-af37-409428048394"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:13.415386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.415335    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9" pod "509e10d8-e99f-4500-af37-409428048394" (UID: "509e10d8-e99f-4500-af37-409428048394"). InnerVolumeSpecName "kube-api-access-blwv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:13.514750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.514667    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509e10d8-e99f-4500-af37-409428048394-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:13.514750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.514699    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509e10d8-e99f-4500-af37-409428048394-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:13.514750 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:13.514709    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwv9\" (UniqueName: \"kubernetes.io/projected/509e10d8-e99f-4500-af37-409428048394-kube-api-access-blwv9\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:14.243146 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:14.243112    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"
Oct 30 20:31:14.243344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:14.243151    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt" event={"ID":"509e10d8-e99f-4500-af37-409428048394","Type":"ContainerDied","Data":"fa3e1cc6ea1e58fd454fba8b90f1759e54352cff7098ee006e6fa6b4b02a5104"}
Oct 30 20:31:14.243344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:14.243210    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa3e1cc6ea1e58fd454fba8b90f1759e54352cff7098ee006e6fa6b4b02a5104"
Oct 30 20:31:15.052198 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:15.052160    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"]
Oct 30 20:31:15.103091 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:15.103057    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3ac03c-gcllq"]
Oct 30 20:31:15.875842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:15.875793    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx" podUID="af57283a-9f6d-4e48-8977-15e91aa67e98" containerName="payload-pre-processing" containerID="cri-o://226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55" gracePeriod=25
Oct 30 20:31:16.001086 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.001059    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf11604b-818d-480b-b2de-ef509defff71" path="/var/lib/kubelet/pods/cf11604b-818d-480b-b2de-ef509defff71/volumes"
Oct 30 20:31:16.138367 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.138295    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 20:31:16.242166 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.242127    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume\") pod \"af57283a-9f6d-4e48-8977-15e91aa67e98\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") "
Oct 30 20:31:16.242432 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.242181    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm\" (UniqueName: \"kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm\") pod \"af57283a-9f6d-4e48-8977-15e91aa67e98\" (UID: \"af57283a-9f6d-4e48-8977-15e91aa67e98\") "
Oct 30 20:31:16.242534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.242514    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume" pod "af57283a-9f6d-4e48-8977-15e91aa67e98" (UID: "af57283a-9f6d-4e48-8977-15e91aa67e98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:16.244369 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.244350    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm" pod "af57283a-9f6d-4e48-8977-15e91aa67e98" (UID: "af57283a-9f6d-4e48-8977-15e91aa67e98"). InnerVolumeSpecName "kube-api-access-4qkcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:16.256484 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.256448    4917 generic.go:356] "Generic (PLEG): container finished" podID="af57283a-9f6d-4e48-8977-15e91aa67e98" containerID="226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55" exitCode=0
Oct 30 20:31:16.256597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.256495    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"
Oct 30 20:31:16.256597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.256528    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx" event={"ID":"af57283a-9f6d-4e48-8977-15e91aa67e98","Type":"ContainerDied","Data":"226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55"}
Oct 30 20:31:16.256597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.256588    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx" event={"ID":"af57283a-9f6d-4e48-8977-15e91aa67e98","Type":"ContainerDied","Data":"e1439a340284a30c98a4e0b3d2ff220d96071490330cb1317675d474c865a27d"}
Oct 30 20:31:16.256709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.256614    4917 scope.go:122] "RemoveContainer" containerID="226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55"
Oct 30 20:31:16.269522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.269497    4917 scope.go:122] "RemoveContainer" containerID="226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55"
Oct 30 20:31:16.269901 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:31:16.269872    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55\": container with ID starting with 226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55 not found: ID does not exist" containerID="226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55"
Oct 30 20:31:16.269967 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.269917    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55"} err="failed to get container status \"226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55\": rpc error: code = NotFound desc = could not find container \"226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55\": container with ID starting with 226201dbbfd0f206ccf7ca8191cd1c97edd1f565b39f984f59fa9946713a8c55 not found: ID does not exist"
Oct 30 20:31:16.344029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.343989    4917 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57283a-9f6d-4e48-8977-15e91aa67e98-plugins-config-volume\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:16.344029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.344020    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qkcm\" (UniqueName: \"kubernetes.io/projected/af57283a-9f6d-4e48-8977-15e91aa67e98-kube-api-access-4qkcm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:16.427864 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.427833    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"]
Oct 30 20:31:16.433788 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:16.433749    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3ccc9f-57584c9f9-g6nmx"]
Oct 30 20:31:18.001881 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:18.001844    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af57283a-9f6d-4e48-8977-15e91aa67e98" path="/var/lib/kubelet/pods/af57283a-9f6d-4e48-8977-15e91aa67e98/volumes"
Oct 30 20:31:18.754490 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:18.754423    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" containerID="cri-o://89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85" gracePeriod=45
Oct 30 20:31:18.805330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:18.805287    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main" containerID="cri-o://6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53" gracePeriod=45
Oct 30 20:31:18.806817 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:18.806784    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main" containerID="cri-o://d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550" gracePeriod=45
Oct 30 20:31:19.064947 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.064916    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:31:19.106461 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.106432    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:31:19.109483 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.109461    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:31:19.173525 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173495    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2\" (UniqueName: \"kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2\") pod \"cb1b3140-5033-4b55-be7e-ea71cf750811\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") "
Oct 30 20:31:19.173723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173551    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache\") pod \"99b5de93-7a58-48c2-ae9a-63483ec8265b\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") "
Oct 30 20:31:19.173723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173572    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir\") pod \"2fae092c-192f-4272-9edc-d46a060463b3\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") "
Oct 30 20:31:19.173723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173594    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm\") pod \"2fae092c-192f-4272-9edc-d46a060463b3\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") "
Oct 30 20:31:19.173723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173638    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm\") pod \"99b5de93-7a58-48c2-ae9a-63483ec8265b\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") "
Oct 30 20:31:19.173723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173678    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb\" (UniqueName: \"kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb\") pod \"2fae092c-192f-4272-9edc-d46a060463b3\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") "
Oct 30 20:31:19.173723 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173723    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs\") pod \"cb1b3140-5033-4b55-be7e-ea71cf750811\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173762    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs\" (UniqueName: \"kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs\") pod \"99b5de93-7a58-48c2-ae9a-63483ec8265b\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173822    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache\") pod \"2fae092c-192f-4272-9edc-d46a060463b3\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173854    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home\") pod \"99b5de93-7a58-48c2-ae9a-63483ec8265b\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173881    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home\") pod \"cb1b3140-5033-4b55-be7e-ea71cf750811\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173887    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir" pod "2fae092c-192f-4272-9edc-d46a060463b3" (UID: "2fae092c-192f-4272-9edc-d46a060463b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173894    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache" pod "99b5de93-7a58-48c2-ae9a-63483ec8265b" (UID: "99b5de93-7a58-48c2-ae9a-63483ec8265b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173915    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache\") pod \"cb1b3140-5033-4b55-be7e-ea71cf750811\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173947    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm\") pod \"cb1b3140-5033-4b55-be7e-ea71cf750811\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.173981    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs\") pod \"2fae092c-192f-4272-9edc-d46a060463b3\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174014    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j\" (UniqueName: \"kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j\") pod \"99b5de93-7a58-48c2-ae9a-63483ec8265b\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") "
Oct 30 20:31:19.174090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174057    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir\") pod \"cb1b3140-5033-4b55-be7e-ea71cf750811\" (UID: \"cb1b3140-5033-4b55-be7e-ea71cf750811\") "
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174110    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir\") pod \"99b5de93-7a58-48c2-ae9a-63483ec8265b\" (UID: \"99b5de93-7a58-48c2-ae9a-63483ec8265b\") "
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174142    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home\") pod \"2fae092c-192f-4272-9edc-d46a060463b3\" (UID: \"2fae092c-192f-4272-9edc-d46a060463b3\") "
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174283    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home" pod "cb1b3140-5033-4b55-be7e-ea71cf750811" (UID: "cb1b3140-5033-4b55-be7e-ea71cf750811"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174492    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home" pod "2fae092c-192f-4272-9edc-d46a060463b3" (UID: "2fae092c-192f-4272-9edc-d46a060463b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174574    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174596    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174612    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.174652 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174627    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.175107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174679    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache" pod "cb1b3140-5033-4b55-be7e-ea71cf750811" (UID: "cb1b3140-5033-4b55-be7e-ea71cf750811"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.175107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.174851    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache" pod "2fae092c-192f-4272-9edc-d46a060463b3" (UID: "2fae092c-192f-4272-9edc-d46a060463b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.175107 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.175051    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home" pod "99b5de93-7a58-48c2-ae9a-63483ec8265b" (UID: "99b5de93-7a58-48c2-ae9a-63483ec8265b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.176054 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.175953    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir" pod "99b5de93-7a58-48c2-ae9a-63483ec8265b" (UID: "99b5de93-7a58-48c2-ae9a-63483ec8265b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.176054 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.176019    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir" pod "cb1b3140-5033-4b55-be7e-ea71cf750811" (UID: "cb1b3140-5033-4b55-be7e-ea71cf750811"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.176678 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.176644    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm" pod "2fae092c-192f-4272-9edc-d46a060463b3" (UID: "2fae092c-192f-4272-9edc-d46a060463b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.176824 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.176734    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs" pod "99b5de93-7a58-48c2-ae9a-63483ec8265b" (UID: "99b5de93-7a58-48c2-ae9a-63483ec8265b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:19.177262 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.177225    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb" pod "2fae092c-192f-4272-9edc-d46a060463b3" (UID: "2fae092c-192f-4272-9edc-d46a060463b3"). InnerVolumeSpecName "kube-api-access-87rpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:19.177361 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.177245    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm" pod "cb1b3140-5033-4b55-be7e-ea71cf750811" (UID: "cb1b3140-5033-4b55-be7e-ea71cf750811"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.177499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.177479    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm" pod "99b5de93-7a58-48c2-ae9a-63483ec8265b" (UID: "99b5de93-7a58-48c2-ae9a-63483ec8265b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.177642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.177617    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs" pod "cb1b3140-5033-4b55-be7e-ea71cf750811" (UID: "cb1b3140-5033-4b55-be7e-ea71cf750811"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:19.178057 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.178038    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2" pod "cb1b3140-5033-4b55-be7e-ea71cf750811" (UID: "cb1b3140-5033-4b55-be7e-ea71cf750811"). InnerVolumeSpecName "kube-api-access-sgwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:19.178368 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.178344    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j" pod "99b5de93-7a58-48c2-ae9a-63483ec8265b" (UID: "99b5de93-7a58-48c2-ae9a-63483ec8265b"). InnerVolumeSpecName "kube-api-access-chd2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:19.178585 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.178567    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs" pod "2fae092c-192f-4272-9edc-d46a060463b3" (UID: "2fae092c-192f-4272-9edc-d46a060463b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275489    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgwd2\" (UniqueName: \"kubernetes.io/projected/cb1b3140-5033-4b55-be7e-ea71cf750811-kube-api-access-sgwd2\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275518    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275527    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275535    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rpb\" (UniqueName: \"kubernetes.io/projected/2fae092c-192f-4272-9edc-d46a060463b3-kube-api-access-87rpb\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275544    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb1b3140-5033-4b55-be7e-ea71cf750811-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275553    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b5de93-7a58-48c2-ae9a-63483ec8265b-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275561    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae092c-192f-4272-9edc-d46a060463b3-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275569    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275578    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275586    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.275588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275594    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae092c-192f-4272-9edc-d46a060463b3-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.276349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275602    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chd2j\" (UniqueName: \"kubernetes.io/projected/99b5de93-7a58-48c2-ae9a-63483ec8265b-kube-api-access-chd2j\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.276349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275610    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb1b3140-5033-4b55-be7e-ea71cf750811-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.276349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.275619    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b5de93-7a58-48c2-ae9a-63483ec8265b-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.282821 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.282789    4917 generic.go:356] "Generic (PLEG): container finished" podID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerID="89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85" exitCode=0
Oct 30 20:31:19.282968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.282825    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" event={"ID":"cb1b3140-5033-4b55-be7e-ea71cf750811","Type":"ContainerDied","Data":"89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85"}
Oct 30 20:31:19.282968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.282855    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"
Oct 30 20:31:19.282968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.282875    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" event={"ID":"cb1b3140-5033-4b55-be7e-ea71cf750811","Type":"ContainerDied","Data":"f8a50e53f1d1454cf4d514d4b74e7520bf7fcd02662733344f85114c2e85005b"}
Oct 30 20:31:19.282968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.282893    4917 scope.go:122] "RemoveContainer" containerID="89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85"
Oct 30 20:31:19.284545 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.284522    4917 generic.go:356] "Generic (PLEG): container finished" podID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerID="6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53" exitCode=0
Oct 30 20:31:19.284687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.284567    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"
Oct 30 20:31:19.284687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.284616    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" event={"ID":"99b5de93-7a58-48c2-ae9a-63483ec8265b","Type":"ContainerDied","Data":"6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53"}
Oct 30 20:31:19.284687 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.284663    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp" event={"ID":"99b5de93-7a58-48c2-ae9a-63483ec8265b","Type":"ContainerDied","Data":"aa9daaf67e3e6f462bf0038c80b1c8fb8e2f5c0932a0750a116ee725cf02a99f"}
Oct 30 20:31:19.286186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.286163    4917 generic.go:356] "Generic (PLEG): container finished" podID="2fae092c-192f-4272-9edc-d46a060463b3" containerID="d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550" exitCode=0
Oct 30 20:31:19.286334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.286212    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" event={"ID":"2fae092c-192f-4272-9edc-d46a060463b3","Type":"ContainerDied","Data":"d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550"}
Oct 30 20:31:19.286334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.286245    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr" event={"ID":"2fae092c-192f-4272-9edc-d46a060463b3","Type":"ContainerDied","Data":"6b837246018d99ba20c73ebbca3490cf7831e5370ef91579f18226df53f5ecb1"}
Oct 30 20:31:19.286334 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.286214    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"
Oct 30 20:31:19.295405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.295345    4917 scope.go:122] "RemoveContainer" containerID="89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85"
Oct 30 20:31:19.296132 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:31:19.296037    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85\": container with ID starting with 89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85 not found: ID does not exist" containerID="89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85"
Oct 30 20:31:19.296132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.296064    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85"} err="failed to get container status \"89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85\": rpc error: code = NotFound desc = could not find container \"89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85\": container with ID starting with 89d91796de95e0d31fa650990e5f666511e815b7f287c4464e9a00ad1d3c7b85 not found: ID does not exist"
Oct 30 20:31:19.296132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.296088    4917 scope.go:122] "RemoveContainer" containerID="6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53"
Oct 30 20:31:19.305041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.305023    4917 scope.go:122] "RemoveContainer" containerID="6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53"
Oct 30 20:31:19.305314 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:31:19.305277    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53\": container with ID starting with 6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53 not found: ID does not exist" containerID="6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53"
Oct 30 20:31:19.305475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.305322    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53"} err="failed to get container status \"6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53\": rpc error: code = NotFound desc = could not find container \"6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53\": container with ID starting with 6a173acba17153f27c5076cd1205bf45bed8f9e75adf11cf2af8288baab69d53 not found: ID does not exist"
Oct 30 20:31:19.305475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.305341    4917 scope.go:122] "RemoveContainer" containerID="d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550"
Oct 30 20:31:19.314986 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.314962    4917 scope.go:122] "RemoveContainer" containerID="d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550"
Oct 30 20:31:19.315303 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:31:19.315281    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550\": container with ID starting with d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550 not found: ID does not exist" containerID="d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550"
Oct 30 20:31:19.315394 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.315312    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550"} err="failed to get container status \"d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550\": rpc error: code = NotFound desc = could not find container \"d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550\": container with ID starting with d9cb872f4e1ab232d08c26a5a656c095fdf7ff6e6508ff068a51d7268178f550 not found: ID does not exist"
Oct 30 20:31:19.539332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.539241    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"]
Oct 30 20:31:19.545007 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.544975    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/premium-w1-3ccc9f-kserve-658f4b8656-rlhqp"]
Oct 30 20:31:19.654720 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.654688    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"]
Oct 30 20:31:19.659568 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.659541    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/simulator-w1-3ccc9f-kserve-6599f58cbf-xzpkr"]
Oct 30 20:31:19.772377 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.772347    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"]
Oct 30 20:31:19.778737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:19.778712    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l"]
Oct 30 20:31:20.001790 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:20.001749    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fae092c-192f-4272-9edc-d46a060463b3" path="/var/lib/kubelet/pods/2fae092c-192f-4272-9edc-d46a060463b3/volumes"
Oct 30 20:31:20.002225 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:20.002209    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" path="/var/lib/kubelet/pods/99b5de93-7a58-48c2-ae9a-63483ec8265b/volumes"
Oct 30 20:31:20.002579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:20.002567    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" path="/var/lib/kubelet/pods/cb1b3140-5033-4b55-be7e-ea71cf750811/volumes"
Oct 30 20:31:20.049028 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:20.048981    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-2-w1-3ccc9f-kserve-5bdccc9688-jnh8l" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 20:31:25.265658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:25.265618    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:31:39.679986 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:31:39.679950    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:31:42.278753 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:42.278715    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:31:44.428171 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:44.428144    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:31:44.493898 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:31:44.493869    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:32:09.683159 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:32:09.683119    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:32:17.007521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007478    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"]
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007872    4917 cpu_manager.go:414] "containerMap: removing container" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007884    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007902    4917 cpu_manager.go:414] "containerMap: removing container" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007907    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377bbb6e-4fb5-47b4-be12-995d2dd14f89" containerName="istio-proxy"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007919    4917 cpu_manager.go:414] "containerMap: removing container" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007924    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007932    4917 cpu_manager.go:414] "containerMap: removing container" podUID="af57283a-9f6d-4e48-8977-15e91aa67e98" containerName="payload-pre-processing"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007937    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af57283a-9f6d-4e48-8977-15e91aa67e98" containerName="payload-pre-processing"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007952    4917 cpu_manager.go:414] "containerMap: removing container" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007957    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007963    4917 cpu_manager.go:414] "containerMap: removing container" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main"
Oct 30 20:32:17.007968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007968    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007979    4917 cpu_manager.go:414] "containerMap: removing container" podUID="509e10d8-e99f-4500-af37-409428048394" containerName="revoke-keys"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.007984    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509e10d8-e99f-4500-af37-409428048394" containerName="revoke-keys"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008047    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1b3140-5033-4b55-be7e-ea71cf750811" containerName="main"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008055    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="af57283a-9f6d-4e48-8977-15e91aa67e98" containerName="payload-pre-processing"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008066    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b5de93-7a58-48c2-ae9a-63483ec8265b" containerName="main"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008072    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fae092c-192f-4272-9edc-d46a060463b3" containerName="main"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008078    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="509e10d8-e99f-4500-af37-409428048394" containerName="revoke-keys"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008083    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b18eed-71c6-49ea-87f9-561fe91dcd22" containerName="cleanup"
Oct 30 20:32:17.008364 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.008089    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="60addf8c-add3-4bad-bd45-6596ce7cf5ce" containerName="cleanup"
Oct 30 20:32:17.011131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.011113    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095098    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e680d509-ed58-4150-bcdb-608c9dc0f244-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095132 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095143    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095234    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095359    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxg4\" (UniqueName: \"kubernetes.io/projected/e680d509-ed58-4150-bcdb-608c9dc0f244-kube-api-access-2kxg4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095405    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095477    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095503    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095549    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.095737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.095640    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.117444 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.117407    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"]
Oct 30 20:32:17.196708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196674    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196708 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196712    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196733    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196793    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196831    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e680d509-ed58-4150-bcdb-608c9dc0f244-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196856    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196880    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.196984 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.196903    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxg4\" (UniqueName: \"kubernetes.io/projected/e680d509-ed58-4150-bcdb-608c9dc0f244-kube-api-access-2kxg4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.197326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.197118    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.197326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.197190    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.197326 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.197264    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.197546 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.197522    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.197621 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.197529    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.197677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.197646    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e680d509-ed58-4150-bcdb-608c9dc0f244-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.199243 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.199214    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.199532 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.199508    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.258692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.258619    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e680d509-ed58-4150-bcdb-608c9dc0f244-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.258940 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.258921    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxg4\" (UniqueName: \"kubernetes.io/projected/e680d509-ed58-4150-bcdb-608c9dc0f244-kube-api-access-2kxg4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fc6zk\" (UID: \"e680d509-ed58-4150-bcdb-608c9dc0f244\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.323920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.323883    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:17.751801 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.751760    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"]
Oct 30 20:32:17.753233 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:32:17.753202    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode680d509_ed58_4150_bcdb_608c9dc0f244.slice/crio-185c5678f52e104fbaee0543bfeb07a21efe0c438859a452a51fbfe28005a726 WatchSource:0}: Error finding container 185c5678f52e104fbaee0543bfeb07a21efe0c438859a452a51fbfe28005a726: Status 404 returned error can't find the container with id 185c5678f52e104fbaee0543bfeb07a21efe0c438859a452a51fbfe28005a726
Oct 30 20:32:17.755368 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.755337    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:32:17.755435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.755404    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:32:17.755435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:17.755431    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:32:18.621842 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:18.621801    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" event={"ID":"e680d509-ed58-4150-bcdb-608c9dc0f244","Type":"ContainerStarted","Data":"096efa375babffb99a1a21d4b23f05401d1f321a8433cf6490147dcd8f02a96c"}
Oct 30 20:32:18.622322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:18.621866    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" event={"ID":"e680d509-ed58-4150-bcdb-608c9dc0f244","Type":"ContainerStarted","Data":"185c5678f52e104fbaee0543bfeb07a21efe0c438859a452a51fbfe28005a726"}
Oct 30 20:32:18.860794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:18.858659    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podStartSLOduration=2.8586261200000003 podStartE2EDuration="2.85862612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:18.79506526 +0000 UTC m=+3034.959466905" watchObservedRunningTime="2026-10-30 20:32:18.85862612 +0000 UTC m=+3035.023027763"
Oct 30 20:32:19.324465 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:19.324423    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:19.326101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:19.326071    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:32:19.326231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:19.326123    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 20:32:20.324903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:20.324868    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:32:20.325320 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:20.324927    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 20:32:21.324991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:21.324954    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:32:21.325402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:21.325016    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 20:32:22.324854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:22.324816    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:32:22.325056 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:22.324879    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 20:32:23.324857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:23.324820    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:32:23.325046 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:23.324875    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 20:32:24.324847 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:24.324811    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 20:32:24.325286 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:24.324868    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 20:32:26.105992 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:26.105953    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:26.106500 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:26.106376    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:26.107276 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:26.107255    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk"
Oct 30 20:32:39.679481 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:32:39.679438    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:32:46.133422 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:46.133391    4917 scope.go:122] "RemoveContainer" containerID="6bbff44922d009ba381ec31c7bd06c671ab37ed6a9af920f6075e9ceacb57871"
Oct 30 20:32:46.142475 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:46.142452    4917 scope.go:122] "RemoveContainer" containerID="61f07b3457204088348b7c5d45e073d847505838157bc352700bc6a99e163ffc"
Oct 30 20:32:54.054388 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:54.054353    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"]
Oct 30 20:32:54.104517 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:54.104484    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0000fc-m5q2v"]
Oct 30 20:32:55.000482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:55.000449    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:32:56.003124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:32:56.003086    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670fdaef-751f-488c-afb7-2f7ab4e81153" path="/var/lib/kubelet/pods/670fdaef-751f-488c-afb7-2f7ab4e81153/volumes"
Oct 30 20:33:09.679290 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:33:09.679247    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:33:16.680179 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:16.680137    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:33:17.639182 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.639135    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"]
Oct 30 20:33:17.642890 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.642864    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.749420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.749382    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"]
Oct 30 20:33:17.782213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782181    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782213 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782218    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6n7\" (UniqueName: \"kubernetes.io/projected/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-kube-api-access-7g6n7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782342    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782392    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782461    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782492    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782510    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782615 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782548    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.782704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.782683    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884303    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884372    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884403    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6n7\" (UniqueName: \"kubernetes.io/projected/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-kube-api-access-7g6n7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884478    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884575 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884506    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884564    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884621    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884654    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884680    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884820    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.884856 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.884842    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.885747 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.885719    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.885977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.885896    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.885977 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.885970    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.887958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.887930    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.888246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.888224    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.948630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.948592    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.949214 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.949191    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6n7\" (UniqueName: \"kubernetes.io/projected/787192ab-bdfa-4cbc-9b85-bc2eaad7f101-kube-api-access-7g6n7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2zdhm\" (UID: \"787192ab-bdfa-4cbc-9b85-bc2eaad7f101\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:17.960048 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:17.960019    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:18.399914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:18.399820    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"]
Oct 30 20:33:18.404178 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:33:18.404136    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787192ab_bdfa_4cbc_9b85_bc2eaad7f101.slice/crio-260ef6e47dc3d03f7869cbf4c0323305e89b606969f34e3064fe6528c7dda59a WatchSource:0}: Error finding container 260ef6e47dc3d03f7869cbf4c0323305e89b606969f34e3064fe6528c7dda59a: Status 404 returned error can't find the container with id 260ef6e47dc3d03f7869cbf4c0323305e89b606969f34e3064fe6528c7dda59a
Oct 30 20:33:18.407232 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:18.407190    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:18.407354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:18.407287    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:18.407354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:18.407335    4917 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:33:18.992574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:18.992532    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" event={"ID":"787192ab-bdfa-4cbc-9b85-bc2eaad7f101","Type":"ContainerStarted","Data":"3f25bc187ab6952e46372c0d60d56a18ba0cf17724ae06385de2575efdcbb0f1"}
Oct 30 20:33:18.992574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:18.992585    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" event={"ID":"787192ab-bdfa-4cbc-9b85-bc2eaad7f101","Type":"ContainerStarted","Data":"260ef6e47dc3d03f7869cbf4c0323305e89b606969f34e3064fe6528c7dda59a"}
Oct 30 20:33:19.139136 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:19.139080    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podStartSLOduration=2.139064686 podStartE2EDuration="2.139064686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:19.138427538 +0000 UTC m=+3095.302829179" watchObservedRunningTime="2026-10-30 20:33:19.139064686 +0000 UTC m=+3095.303466327"
Oct 30 20:33:19.962794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:19.962379    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:19.962794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:19.962558    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:19.962794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:19.962617    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:20.966080 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:20.966034    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:20.966623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:20.966106    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:21.961253 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:21.961208    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:21.961594 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:21.961562    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:22.961222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:22.961178    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:22.961897 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:22.961863    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:23.966348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:23.966238    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:23.966348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:23.966329    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:24.961074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:24.961032    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:24.961317 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:24.961130    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:25.960869 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:25.960815    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 20:33:25.961560 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:25.961526    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 20:33:27.962349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:27.962308    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:27.962827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:27.962373    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:29.003962 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:29.003911    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:29.004502 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:29.003984    4917 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:29.189682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:29.189644    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:29.190005 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:29.189976    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:29.190682 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:29.190663    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm"
Oct 30 20:33:29.966738 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:29.966700    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:33:39.679623 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:33:39.679582    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:33:46.051328 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:46.051276    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"]
Oct 30 20:33:46.101574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:46.101539    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab9a460d-j5jcm"]
Oct 30 20:33:46.167920 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:46.167891    4917 scope.go:122] "RemoveContainer" containerID="dcbecc95e7c95e4ef2eb74c6c9ac34201720a7dedc7a707fb7ed56717e62e099"
Oct 30 20:33:46.177389 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:46.177365    4917 scope.go:122] "RemoveContainer" containerID="9a8010565ba9bfd4ef8bb543b29d0a37da796b32570332401627d67fafc8d8d9"
Oct 30 20:33:48.001252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:48.001218    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f07688-df10-48c8-bb20-b9756a4a6d9e" path="/var/lib/kubelet/pods/66f07688-df10-48c8-bb20-b9756a4a6d9e/volumes"
Oct 30 20:33:51.052788 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:51.052748    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"]
Oct 30 20:33:51.103400 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:51.103366    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c700f5-nnjlw"]
Oct 30 20:33:51.409906 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:51.409819    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:33:52.004854 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:33:52.004814    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddeb808-ae9b-4b13-b985-a4268a27492f" path="/var/lib/kubelet/pods/0ddeb808-ae9b-4b13-b985-a4268a27492f/volumes"
Oct 30 20:34:09.679483 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:34:09.679444    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:34:09.771618 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:09.771584    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:34:12.158794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:12.154885    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:34:12.158794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:12.154976    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fc6zk" podUID="e680d509-ed58-4150-bcdb-608c9dc0f244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:34:17.052036 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:17.052000    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"]
Oct 30 20:34:17.103988 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:17.103947    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cef4a9c-j4n9v"]
Oct 30 20:34:18.001583 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:18.001540    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f352e693-eead-442d-92db-0068374d8216" path="/var/lib/kubelet/pods/f352e693-eead-442d-92db-0068374d8216/volumes"
Oct 30 20:34:39.679065 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:34:39.679025    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:34:42.601660 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:42.601622    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:34:45.234942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:45.234892    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:34:45.235451 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:45.234978    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2zdhm" podUID="787192ab-bdfa-4cbc-9b85-bc2eaad7f101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:34:46.207106 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:46.207069    4917 scope.go:122] "RemoveContainer" containerID="86018f243cea93d51f16ad01147fcdff5c922f637fae43973817f9c5df057b8f"
Oct 30 20:34:46.216440 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:46.216420    4917 scope.go:122] "RemoveContainer" containerID="0cd2958d0011766899719e5e9bf87ac39df9e4fc1cd0d767682585c04e6d2622"
Oct 30 20:34:46.225123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:46.225101    4917 scope.go:122] "RemoveContainer" containerID="43c5c1082a7b7d9b51c50660b75b9ea7b059ed2c81d45c0c4291b37215bdfd01"
Oct 30 20:34:46.233358 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:46.233339    4917 scope.go:122] "RemoveContainer" containerID="b74dcc8d7006078191d6d65bb57e5f6484f5ededdb68ebb40b133414c3951556"
Oct 30 20:34:53.304520 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:34:53.304479    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:35:05.502885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:05.502850    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:35:06.051478 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:06.051439    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"]
Oct 30 20:35:06.101786 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:06.101733    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de3fef-wktbf"]
Oct 30 20:35:08.000970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:08.000933    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ef6971-4b73-4d6b-bfba-1f75681fe4fb" path="/var/lib/kubelet/pods/01ef6971-4b73-4d6b-bfba-1f75681fe4fb/volumes"
Oct 30 20:35:09.679534 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:35:09.679500    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:35:28.308515 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:28.308478    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:35:37.873255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:37.873215    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:35:39.678910 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:35:39.678874    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:35:46.265700 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:46.265665    4917 scope.go:122] "RemoveContainer" containerID="86ffcd5040424fea30386492af9a4fdc5f6cb5739ad5fb6a101d6e163397d602"
Oct 30 20:35:46.274565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:46.274540    4917 scope.go:122] "RemoveContainer" containerID="2f1d5424b7d657bf3d95dd76a775664f7045f73ca4e96b5424204d964aaaf6b4"
Oct 30 20:35:46.282295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:46.282272    4917 scope.go:122] "RemoveContainer" containerID="a1a412b9f40fd76624a157e0fbf3df128c45bce4d34b9b8606497f576804f1de"
Oct 30 20:35:59.866336 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:35:59.866300    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:36:09.678545 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:36:09.678508    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:36:12.172033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:12.171996    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:36:14.051605 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:14.051571    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"]
Oct 30 20:36:14.102365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:14.102331    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff41f2ef-dflqt"]
Oct 30 20:36:16.001209 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:16.001175    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509e10d8-e99f-4500-af37-409428048394" path="/var/lib/kubelet/pods/509e10d8-e99f-4500-af37-409428048394/volumes"
Oct 30 20:36:32.177802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:32.177694    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:36:34.575926 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:34.575881    4917 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-kt69c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:36:34.576571 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:34.576533    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-kt69c" podUID="b17447df-4223-406e-8271-9c6d9b5e15a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:36:39.678958 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:36:39.678926    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:36:44.462128 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:44.462100    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:36:44.532579 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:44.532549    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:36:46.313318 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:46.313283    4917 scope.go:122] "RemoveContainer" containerID="95c46c03515c56f81d0cb5476bc8538ae1908d0e2075624f61c912c8906c4252"
Oct 30 20:36:54.262781 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:36:54.262725    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:37:07.488405 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:37:07.485190    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:37:09.734561 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:37:09.734520    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:37:16.074751 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:37:16.074712    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:37:38.372969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:37:38.372929    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:37:39.678900 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:37:39.678863    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:37:46.333435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:37:46.333402    4917 scope.go:122] "RemoveContainer" containerID="f67df3be59c26e1aef6479129653dd2c643048b0f6debe72dcd046cad2d2d08a"
Oct 30 20:37:48.267882 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:37:48.267843    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:38:09.678997 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:38:09.678959    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:38:13.112887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:38:13.112851    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:38:22.990053 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:38:22.990009    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:38:39.678855 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:38:39.678813    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:38:50.795639 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:38:50.795597    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:39:06.048507 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.048473    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"]
Oct 30 20:39:06.052101 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.052052    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.124709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.124673    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.124896 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.124727    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.124896 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.124868    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfs5r\" (UniqueName: \"kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.124991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.124909    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.124991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.124936    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.125066 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.125038    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.156504 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.156474    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b2cfa0\"/\"default-dockercfg-kcs85\""
Oct 30 20:39:06.156661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.156562    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b2cfa0\"/\"simulator-single-main-b2cfa0-kserve-self-signed-certs\""
Oct 30 20:39:06.156661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.156615    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b2cfa0\"/\"kube-root-ca.crt\""
Oct 30 20:39:06.156661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.156614    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b2cfa0\"/\"openshift-service-ca.crt\""
Oct 30 20:39:06.158752 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.158731    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"]
Oct 30 20:39:06.225516 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225477    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.225717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225525    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs5r\" (UniqueName: \"kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.225717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225554    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.225717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225575    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.225717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225615    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.225717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225654    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.226029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.225995    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.226029 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.226024    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.226115 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.226094    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.227972 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.227954    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.228250 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.228234    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.292434 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.292402    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs5r\" (UniqueName: \"kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r\") pod \"simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") " pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.363638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.363543    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:06.753877 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.753845    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"]
Oct 30 20:39:06.755662 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:39:06.755630    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e9f91a1_4068_4bdf_a37b_62b2a18f46b2.slice/crio-c66ff01999051d364468d04ea799bdf0711d13fbe25df93b37eaef8c2ab8f552 WatchSource:0}: Error finding container c66ff01999051d364468d04ea799bdf0711d13fbe25df93b37eaef8c2ab8f552: Status 404 returned error can't find the container with id c66ff01999051d364468d04ea799bdf0711d13fbe25df93b37eaef8c2ab8f552
Oct 30 20:39:06.894585 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.894549    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" event={"ID":"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2","Type":"ContainerStarted","Data":"2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111"}
Oct 30 20:39:06.894764 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:06.894592    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" event={"ID":"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2","Type":"ContainerStarted","Data":"c66ff01999051d364468d04ea799bdf0711d13fbe25df93b37eaef8c2ab8f552"}
Oct 30 20:39:07.016524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:07.016424    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" podStartSLOduration=2.01641159 podStartE2EDuration="2.01641159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:07.014303492 +0000 UTC m=+3443.178705156" watchObservedRunningTime="2026-10-30 20:39:07.01641159 +0000 UTC m=+3443.180813228"
Oct 30 20:39:07.364588 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:07.364520    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:09.679380 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:39:09.679342    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:39:10.639381 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:10.639343    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:39:16.364398 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:16.364361    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:16.380845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:16.380821    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:16.961827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:16.961798    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:39:29.481713 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:29.481638    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:39:36.774513 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.774473    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"]
Oct 30 20:39:36.778130 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.778111    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:36.837492 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.837456    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"]
Oct 30 20:39:36.884613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.884578    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:39:36.884613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.884600    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:39:36.884613 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.884600    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:39:36.884895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.884690    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:39:36.899021 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.898995    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:36.899160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.899142    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:36.899222 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.899204    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6d4\" (UniqueName: \"kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:36.999885 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.999850    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.000074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.999889    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6d4\" (UniqueName: \"kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.000074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:36.999939    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.000717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:37.000693    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.002359 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:37.002339    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.063598 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:37.063528    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6d4\" (UniqueName: \"kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4\") pod \"maas-api-revoke-sub-942f676e-kmnxz\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.091188 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:37.091163    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:37.468657 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:37.468629    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"]
Oct 30 20:39:37.471075 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:39:37.471040    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5378191b_ef74_49c6_9bcb_5ee41ec9147e.slice/crio-d8683d63d3d170d82dac6dfcb9bb3ebabc3348c898e6e4547748c40d979ac64e WatchSource:0}: Error finding container d8683d63d3d170d82dac6dfcb9bb3ebabc3348c898e6e4547748c40d979ac64e: Status 404 returned error can't find the container with id d8683d63d3d170d82dac6dfcb9bb3ebabc3348c898e6e4547748c40d979ac64e
Oct 30 20:39:38.055432 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:38.055344    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz" event={"ID":"5378191b-ef74-49c6-9bcb-5ee41ec9147e","Type":"ContainerStarted","Data":"9633e3feaac92be3c9d0582a1a5122a7b849eaf3b05a0a536101ccc53688791d"}
Oct 30 20:39:38.055432 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:38.055403    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz" event={"ID":"5378191b-ef74-49c6-9bcb-5ee41ec9147e","Type":"ContainerStarted","Data":"d8683d63d3d170d82dac6dfcb9bb3ebabc3348c898e6e4547748c40d979ac64e"}
Oct 30 20:39:38.191732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:38.191674    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz" podStartSLOduration=2.191658839 podStartE2EDuration="2.191658839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:38.189763011 +0000 UTC m=+3474.354164655" watchObservedRunningTime="2026-10-30 20:39:38.191658839 +0000 UTC m=+3474.356060480"
Oct 30 20:39:39.061965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:39.061932    4917 generic.go:356] "Generic (PLEG): container finished" podID="5378191b-ef74-49c6-9bcb-5ee41ec9147e" containerID="9633e3feaac92be3c9d0582a1a5122a7b849eaf3b05a0a536101ccc53688791d" exitCode=0
Oct 30 20:39:39.062422 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:39.062020    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz" event={"ID":"5378191b-ef74-49c6-9bcb-5ee41ec9147e","Type":"ContainerDied","Data":"9633e3feaac92be3c9d0582a1a5122a7b849eaf3b05a0a536101ccc53688791d"}
Oct 30 20:39:39.679066 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:39:39.679024    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:39:40.202857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.202827    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:40.332215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.332142    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca\") pod \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") "
Oct 30 20:39:40.332215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.332181    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca\") pod \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") "
Oct 30 20:39:40.332418 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.332336    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4\" (UniqueName: \"kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4\") pod \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\" (UID: \"5378191b-ef74-49c6-9bcb-5ee41ec9147e\") "
Oct 30 20:39:40.332499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.332477    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca" pod "5378191b-ef74-49c6-9bcb-5ee41ec9147e" (UID: "5378191b-ef74-49c6-9bcb-5ee41ec9147e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:40.332693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.332669    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:40.334456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.334429    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca" pod "5378191b-ef74-49c6-9bcb-5ee41ec9147e" (UID: "5378191b-ef74-49c6-9bcb-5ee41ec9147e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:40.334606 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.334583    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4" pod "5378191b-ef74-49c6-9bcb-5ee41ec9147e" (UID: "5378191b-ef74-49c6-9bcb-5ee41ec9147e"). InnerVolumeSpecName "kube-api-access-7h6d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:40.433474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.433442    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6d4\" (UniqueName: \"kubernetes.io/projected/5378191b-ef74-49c6-9bcb-5ee41ec9147e-kube-api-access-7h6d4\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:40.433474 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:40.433479    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5378191b-ef74-49c6-9bcb-5ee41ec9147e-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:41.073754 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:41.073709    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz" event={"ID":"5378191b-ef74-49c6-9bcb-5ee41ec9147e","Type":"ContainerDied","Data":"d8683d63d3d170d82dac6dfcb9bb3ebabc3348c898e6e4547748c40d979ac64e"}
Oct 30 20:39:41.073966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:41.073784    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8683d63d3d170d82dac6dfcb9bb3ebabc3348c898e6e4547748c40d979ac64e"
Oct 30 20:39:41.073966 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:41.073725    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"
Oct 30 20:39:49.018499 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.018462    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"]
Oct 30 20:39:49.018935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.018860    4917 cpu_manager.go:414] "containerMap: removing container" podUID="5378191b-ef74-49c6-9bcb-5ee41ec9147e" containerName="revoke-keys"
Oct 30 20:39:49.018935 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.018871    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5378191b-ef74-49c6-9bcb-5ee41ec9147e" containerName="revoke-keys"
Oct 30 20:39:49.019018 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.018949    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="5378191b-ef74-49c6-9bcb-5ee41ec9147e" containerName="revoke-keys"
Oct 30 20:39:49.021970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.021953    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.078969 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.078940    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:39:49.079140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.078976    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:39:49.079219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.079160    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:39:49.085975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.079197    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:39:49.114336 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.114300    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.114336 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.114337    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.114551 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.114454    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwkp\" (UniqueName: \"kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.132013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.131980    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"]
Oct 30 20:39:49.215630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.215597    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.215828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.215635    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.215828 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.215709    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwkp\" (UniqueName: \"kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.216342 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.216319    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.218122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.218098    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.279366 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.279298    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwkp\" (UniqueName: \"kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.334658 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.334628    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:49.710043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.710012    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"]
Oct 30 20:39:49.711416 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:39:49.711383    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d3298b_f972_498e_b816_b937941c3a6c.slice/crio-e0589f6f5ac6454decc215822608da018ed4ab31e348a158f70e01a07f14ee0b WatchSource:0}: Error finding container e0589f6f5ac6454decc215822608da018ed4ab31e348a158f70e01a07f14ee0b: Status 404 returned error can't find the container with id e0589f6f5ac6454decc215822608da018ed4ab31e348a158f70e01a07f14ee0b
Oct 30 20:39:49.714246 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:49.714225    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:39:50.119437 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:50.119404    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m" event={"ID":"82d3298b-f972-498e-b816-b937941c3a6c","Type":"ContainerStarted","Data":"86d4602856ed9f00f9041163b2ed41065f57da5a66e750145b95be27ac756bea"}
Oct 30 20:39:50.119873 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:50.119455    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m" event={"ID":"82d3298b-f972-498e-b816-b937941c3a6c","Type":"ContainerStarted","Data":"e0589f6f5ac6454decc215822608da018ed4ab31e348a158f70e01a07f14ee0b"}
Oct 30 20:39:50.245459 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:50.245362    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m" podStartSLOduration=2.245348394 podStartE2EDuration="2.245348394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:50.243929856 +0000 UTC m=+3486.408331497" watchObservedRunningTime="2026-10-30 20:39:50.245348394 +0000 UTC m=+3486.409750036"
Oct 30 20:39:51.126344 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:51.126311    4917 generic.go:356] "Generic (PLEG): container finished" podID="82d3298b-f972-498e-b816-b937941c3a6c" containerID="86d4602856ed9f00f9041163b2ed41065f57da5a66e750145b95be27ac756bea" exitCode=0
Oct 30 20:39:51.126756 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:51.126365    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m" event={"ID":"82d3298b-f972-498e-b816-b937941c3a6c","Type":"ContainerDied","Data":"86d4602856ed9f00f9041163b2ed41065f57da5a66e750145b95be27ac756bea"}
Oct 30 20:39:52.270870 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.270848    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:52.444694 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.444655    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca\") pod \"82d3298b-f972-498e-b816-b937941c3a6c\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") "
Oct 30 20:39:52.444915 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.444708    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp\" (UniqueName: \"kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp\") pod \"82d3298b-f972-498e-b816-b937941c3a6c\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") "
Oct 30 20:39:52.444915 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.444890    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca\") pod \"82d3298b-f972-498e-b816-b937941c3a6c\" (UID: \"82d3298b-f972-498e-b816-b937941c3a6c\") "
Oct 30 20:39:52.445252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.445227    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca" pod "82d3298b-f972-498e-b816-b937941c3a6c" (UID: "82d3298b-f972-498e-b816-b937941c3a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:52.446968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.446942    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca" pod "82d3298b-f972-498e-b816-b937941c3a6c" (UID: "82d3298b-f972-498e-b816-b937941c3a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.447074 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.447050    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp" pod "82d3298b-f972-498e-b816-b937941c3a6c" (UID: "82d3298b-f972-498e-b816-b937941c3a6c"). InnerVolumeSpecName "kube-api-access-xxwkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:52.546124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.546090    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.546124 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.546122    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwkp\" (UniqueName: \"kubernetes.io/projected/82d3298b-f972-498e-b816-b937941c3a6c-kube-api-access-xxwkp\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.546336 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:52.546134    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82d3298b-f972-498e-b816-b937941c3a6c-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:53.138717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:53.138683    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"
Oct 30 20:39:53.138717 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:53.138691    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m" event={"ID":"82d3298b-f972-498e-b816-b937941c3a6c","Type":"ContainerDied","Data":"e0589f6f5ac6454decc215822608da018ed4ab31e348a158f70e01a07f14ee0b"}
Oct 30 20:39:53.138975 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:53.138742    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0589f6f5ac6454decc215822608da018ed4ab31e348a158f70e01a07f14ee0b"
Oct 30 20:39:53.654460 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:39:53.654424    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"]
Oct 30 20:40:06.783601 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:06.783555    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:40:08.674718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.674674    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" podUID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" containerName="main" containerID="cri-o://2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111" gracePeriod=45
Oct 30 20:40:08.954862 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.954840    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:40:08.987014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.986984    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs\") pod \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") "
Oct 30 20:40:08.987159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987028    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache\") pod \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") "
Oct 30 20:40:08.987159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987052    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm\") pod \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") "
Oct 30 20:40:08.987159 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987132    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir\") pod \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") "
Oct 30 20:40:08.987284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987157    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r\" (UniqueName: \"kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r\") pod \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") "
Oct 30 20:40:08.987284 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987184    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home\") pod \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\" (UID: \"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2\") "
Oct 30 20:40:08.987386 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987302    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache" pod "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" (UID: "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:08.987453 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987387    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir" pod "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" (UID: "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:08.987574 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987550    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home" pod "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" (UID: "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:08.987647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987556    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:08.987647 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.987606    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:08.989309 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.989287    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm" pod "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" (UID: "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:08.989436 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.989412    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs" pod "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" (UID: "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:08.989476 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:08.989460    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r" pod "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" (UID: "8e9f91a1-4068-4bdf-a37b-62b2a18f46b2"). InnerVolumeSpecName "kube-api-access-mfs5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:09.088231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.088177    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfs5r\" (UniqueName: \"kubernetes.io/projected/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-kube-api-access-mfs5r\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:09.088231 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.088236    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:09.088446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.088251    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:09.088446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.088264    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:09.234709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.234621    4917 generic.go:356] "Generic (PLEG): container finished" podID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" containerID="2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111" exitCode=0
Oct 30 20:40:09.234709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.234674    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"
Oct 30 20:40:09.235024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.234698    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" event={"ID":"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2","Type":"ContainerDied","Data":"2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111"}
Oct 30 20:40:09.235024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.234743    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" event={"ID":"8e9f91a1-4068-4bdf-a37b-62b2a18f46b2","Type":"ContainerDied","Data":"c66ff01999051d364468d04ea799bdf0711d13fbe25df93b37eaef8c2ab8f552"}
Oct 30 20:40:09.235024 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.234761    4917 scope.go:122] "RemoveContainer" containerID="2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111"
Oct 30 20:40:09.244342 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.244321    4917 scope.go:122] "RemoveContainer" containerID="2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111"
Oct 30 20:40:09.244587 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:40:09.244566    4917 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111\": container with ID starting with 2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111 not found: ID does not exist" containerID="2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111"
Oct 30 20:40:09.244655 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.244594    4917 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111"} err="failed to get container status \"2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111\": rpc error: code = NotFound desc = could not find container \"2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111\": container with ID starting with 2e2362899ad69104b0a113ae31dd4620d62c280e2dc3bdf04438f3bb8a588111 not found: ID does not exist"
Oct 30 20:40:09.402709 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.402672    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"]
Oct 30 20:40:09.407373 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.407342    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87"]
Oct 30 20:40:09.679473 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:40:09.679440    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:40:09.951596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:09.951509    4917 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-b2cfa0/simulator-single-main-b2cfa0-kserve-6b8c74585d-qjj87" podUID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.200:8000/ready\": context deadline exceeded"
Oct 30 20:40:10.001355 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:10.001324    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" path="/var/lib/kubelet/pods/8e9f91a1-4068-4bdf-a37b-62b2a18f46b2/volumes"
Oct 30 20:40:10.334061 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:10.333978    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"]
Oct 30 20:40:10.384168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:10.384135    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b2cfa0-drg4m"]
Oct 30 20:40:12.001432 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:40:12.001399    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d3298b-f972-498e-b816-b937941c3a6c" path="/var/lib/kubelet/pods/82d3298b-f972-498e-b816-b937941c3a6c/volumes"
Oct 30 20:40:39.687608 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:40:39.687564    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:41:09.679028 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:41:09.678991    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:41:39.678678 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:41:39.678635    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:41:44.499032 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:41:44.498999    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:41:44.568487 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:41:44.568463    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:42:04.111235 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:42:04.111201    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:42:09.679984 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:42:09.679943    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:42:21.204013 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:42:21.203977    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:42:39.678502 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:42:39.678458    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:43:00.804295 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:00.804255    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:43:09.678645 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:43:09.678607    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:43:16.014610 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.014572    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"]
Oct 30 20:43:16.015041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.014956    4917 cpu_manager.go:414] "containerMap: removing container" podUID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" containerName="main"
Oct 30 20:43:16.015041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.014968    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" containerName="main"
Oct 30 20:43:16.015041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.014987    4917 cpu_manager.go:414] "containerMap: removing container" podUID="82d3298b-f972-498e-b816-b937941c3a6c" containerName="revoke-keys"
Oct 30 20:43:16.015041 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.014993    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d3298b-f972-498e-b816-b937941c3a6c" containerName="revoke-keys"
Oct 30 20:43:16.015205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.015066    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9f91a1-4068-4bdf-a37b-62b2a18f46b2" containerName="main"
Oct 30 20:43:16.015205 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.015076    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d3298b-f972-498e-b816-b937941c3a6c" containerName="revoke-keys"
Oct 30 20:43:16.018265 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.018241    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.076742 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.076705    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.077050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.076760    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.077050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.076939    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.077050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.077007    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.077050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.077048    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.077280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.077077    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wcf\" (UniqueName: \"kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.121958 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.121921    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c530ce\"/\"simulator-single-main-c530ce-kserve-self-signed-certs\""
Oct 30 20:43:16.122177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.122096    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c530ce\"/\"openshift-service-ca.crt\""
Oct 30 20:43:16.122177 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.122136    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c530ce\"/\"default-dockercfg-fcvs4\""
Oct 30 20:43:16.122291 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.122172    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c530ce\"/\"kube-root-ca.crt\""
Oct 30 20:43:16.124356 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.124332    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"]
Oct 30 20:43:16.178111 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178078    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178123    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178159    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178190    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178211    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178229    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wcf\" (UniqueName: \"kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178528    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178627 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178584    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.178703 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.178633    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.180650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.180630    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.180805 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.180786    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.240981 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.240946    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wcf\" (UniqueName: \"kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf\") pod \"simulator-single-main-c530ce-kserve-8676b4765-2dq4n\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") " pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.330039 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.329930    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:16.713332 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:16.713288    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"]
Oct 30 20:43:16.715027 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:43:16.714998    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod660413ec_19a5_4698_8f14_e495c20addaf.slice/crio-807b362c3c12abbdacbf94110e74ec1b6d4f026f7e6d995d5b0e2ed1a84c4108 WatchSource:0}: Error finding container 807b362c3c12abbdacbf94110e74ec1b6d4f026f7e6d995d5b0e2ed1a84c4108: Status 404 returned error can't find the container with id 807b362c3c12abbdacbf94110e74ec1b6d4f026f7e6d995d5b0e2ed1a84c4108
Oct 30 20:43:17.215165 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:17.215124    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n" event={"ID":"660413ec-19a5-4698-8f14-e495c20addaf","Type":"ContainerStarted","Data":"52d77552c08e124dc546b682d9d1a53630a4bb3bb9405f470794df1b9e2952b9"}
Oct 30 20:43:17.215542 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:17.215174    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n" event={"ID":"660413ec-19a5-4698-8f14-e495c20addaf","Type":"ContainerStarted","Data":"807b362c3c12abbdacbf94110e74ec1b6d4f026f7e6d995d5b0e2ed1a84c4108"}
Oct 30 20:43:17.330707 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:17.330670    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:17.380294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:17.380234    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n" podStartSLOduration=2.380218866 podStartE2EDuration="2.380218866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:17.339711298 +0000 UTC m=+3693.504112955" watchObservedRunningTime="2026-10-30 20:43:17.380218866 +0000 UTC m=+3693.544620508"
Oct 30 20:43:20.507294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:20.507244    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:43:26.330168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:26.330124    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:26.342797 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:26.342756    4917 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:27.279930 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:27.279899    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:43:39.678501 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:43:39.678464    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:43:44.460095 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:44.460061    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:43:46.198900 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.198866    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"]
Oct 30 20:43:46.202732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.202714    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.262823 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.262753    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.263014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.262905    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.263014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.262947    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfv8\" (UniqueName: \"kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.306335 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.306296    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:43:46.306335 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.306327    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:43:46.306335 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.306337    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:43:46.306641 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.306300    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:43:46.308661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.308634    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"]
Oct 30 20:43:46.363428 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.363393    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.363603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.363555    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.363603 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.363594    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfv8\" (UniqueName: \"kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.364293 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.364264    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.365961 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.365942    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.425763 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.425735    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfv8\" (UniqueName: \"kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8\") pod \"maas-api-revoke-sub-757f63d0-x2ckq\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.515240 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.515166    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:46.892157 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:46.892128    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"]
Oct 30 20:43:46.893693 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:43:46.893664    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70edaeb_81c7_4079_b319_e4a3008c444a.slice/crio-d5ebb971a3082cde04b246f74984af891458520b45767a5fcdc5ed797eaacab3 WatchSource:0}: Error finding container d5ebb971a3082cde04b246f74984af891458520b45767a5fcdc5ed797eaacab3: Status 404 returned error can't find the container with id d5ebb971a3082cde04b246f74984af891458520b45767a5fcdc5ed797eaacab3
Oct 30 20:43:47.372343 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:47.372304    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq" event={"ID":"a70edaeb-81c7-4079-b319-e4a3008c444a","Type":"ContainerStarted","Data":"0cd06ea0431ae190f4ecb7819712813c5186fd3c670d43aa6a3fbb53740b6185"}
Oct 30 20:43:47.372757 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:47.372368    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq" event={"ID":"a70edaeb-81c7-4079-b319-e4a3008c444a","Type":"ContainerStarted","Data":"d5ebb971a3082cde04b246f74984af891458520b45767a5fcdc5ed797eaacab3"}
Oct 30 20:43:47.493962 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:47.493901    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq" podStartSLOduration=1.493888226 podStartE2EDuration="1.493888226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:47.491742542 +0000 UTC m=+3723.656144187" watchObservedRunningTime="2026-10-30 20:43:47.493888226 +0000 UTC m=+3723.658289909"
Oct 30 20:43:48.378646 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:48.378611    4917 generic.go:356] "Generic (PLEG): container finished" podID="a70edaeb-81c7-4079-b319-e4a3008c444a" containerID="0cd06ea0431ae190f4ecb7819712813c5186fd3c670d43aa6a3fbb53740b6185" exitCode=0
Oct 30 20:43:48.379119 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:48.378695    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq" event={"ID":"a70edaeb-81c7-4079-b319-e4a3008c444a","Type":"ContainerDied","Data":"0cd06ea0431ae190f4ecb7819712813c5186fd3c670d43aa6a3fbb53740b6185"}
Oct 30 20:43:49.518289 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.518261    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:49.594011 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.593981    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8\" (UniqueName: \"kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8\") pod \"a70edaeb-81c7-4079-b319-e4a3008c444a\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") "
Oct 30 20:43:49.594186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.594047    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca\") pod \"a70edaeb-81c7-4079-b319-e4a3008c444a\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") "
Oct 30 20:43:49.594186 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.594073    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca\") pod \"a70edaeb-81c7-4079-b319-e4a3008c444a\" (UID: \"a70edaeb-81c7-4079-b319-e4a3008c444a\") "
Oct 30 20:43:49.594513 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.594483    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca" pod "a70edaeb-81c7-4079-b319-e4a3008c444a" (UID: "a70edaeb-81c7-4079-b319-e4a3008c444a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:49.596216 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.596188    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8" pod "a70edaeb-81c7-4079-b319-e4a3008c444a" (UID: "a70edaeb-81c7-4079-b319-e4a3008c444a"). InnerVolumeSpecName "kube-api-access-6nfv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:49.596322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.596193    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca" pod "a70edaeb-81c7-4079-b319-e4a3008c444a" (UID: "a70edaeb-81c7-4079-b319-e4a3008c444a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:43:49.695597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.695567    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfv8\" (UniqueName: \"kubernetes.io/projected/a70edaeb-81c7-4079-b319-e4a3008c444a-kube-api-access-6nfv8\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:49.695597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.695596    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:49.695838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:49.695606    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70edaeb-81c7-4079-b319-e4a3008c444a-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:50.390337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:50.390255    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"
Oct 30 20:43:50.390337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:50.390283    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq" event={"ID":"a70edaeb-81c7-4079-b319-e4a3008c444a","Type":"ContainerDied","Data":"d5ebb971a3082cde04b246f74984af891458520b45767a5fcdc5ed797eaacab3"}
Oct 30 20:43:50.390337 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:50.390337    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ebb971a3082cde04b246f74984af891458520b45767a5fcdc5ed797eaacab3"
Oct 30 20:43:57.684486 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.684406    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"]
Oct 30 20:43:57.684891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.684820    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a70edaeb-81c7-4079-b319-e4a3008c444a" containerName="revoke-keys"
Oct 30 20:43:57.684891 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.684833    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70edaeb-81c7-4079-b319-e4a3008c444a" containerName="revoke-keys"
Oct 30 20:43:57.684970 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.684905    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70edaeb-81c7-4079-b319-e4a3008c444a" containerName="revoke-keys"
Oct 30 20:43:57.688015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.687995    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.771880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.771842    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hxk\" (UniqueName: \"kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.771880 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.771883    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.772147 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.772028    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.792849 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.792806    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:43:57.793016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.792871    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:43:57.793016 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.792914    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:43:57.795127 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.795105    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"]
Oct 30 20:43:57.846705 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.846663    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:43:57.872645 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.872614    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.872827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.872704    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hxk\" (UniqueName: \"kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.872827 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.872739    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.873496 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.873476    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.875109 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.875088    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:57.935282 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:57.935203    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hxk\" (UniqueName: \"kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:58.000401 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:58.000365    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:43:58.376982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:58.376959    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"]
Oct 30 20:43:58.379202 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:43:58.379171    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ba91f6_02ee_47d7_95fa_3dbc4b61eee6.slice/crio-8efb31ead95e57a1035401f53ed5dff1e78282de95ce43116daebc83b765dbbf WatchSource:0}: Error finding container 8efb31ead95e57a1035401f53ed5dff1e78282de95ce43116daebc83b765dbbf: Status 404 returned error can't find the container with id 8efb31ead95e57a1035401f53ed5dff1e78282de95ce43116daebc83b765dbbf
Oct 30 20:43:58.434589 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:58.434558    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh" event={"ID":"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6","Type":"ContainerStarted","Data":"8efb31ead95e57a1035401f53ed5dff1e78282de95ce43116daebc83b765dbbf"}
Oct 30 20:43:59.203259 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:59.203218    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:43:59.440692 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:59.440660    4917 generic.go:356] "Generic (PLEG): container finished" podID="a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" containerID="8472b265f6bd8a70ccde0a7bf5ae1da8be9267461bff4180439713917a678472" exitCode=0
Oct 30 20:43:59.440887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:43:59.440705    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh" event={"ID":"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6","Type":"ContainerDied","Data":"8472b265f6bd8a70ccde0a7bf5ae1da8be9267461bff4180439713917a678472"}
Oct 30 20:44:00.584528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.584498    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:44:00.700116 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.700022    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca\") pod \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") "
Oct 30 20:44:00.700273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.700185    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca\") pod \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") "
Oct 30 20:44:00.700273 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.700222    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk\" (UniqueName: \"kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk\") pod \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\" (UID: \"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6\") "
Oct 30 20:44:00.700630 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.700597    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca" pod "a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" (UID: "a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:00.702311 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.702283    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca" pod "a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" (UID: "a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:00.702420 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.702346    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk" pod "a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" (UID: "a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6"). InnerVolumeSpecName "kube-api-access-s2hxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:00.801732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.801696    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.801732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.801727    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2hxk\" (UniqueName: \"kubernetes.io/projected/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-kube-api-access-s2hxk\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.801732 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:00.801737    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:01.453402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:01.453369    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"
Oct 30 20:44:01.453402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:01.453390    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh" event={"ID":"a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6","Type":"ContainerDied","Data":"8efb31ead95e57a1035401f53ed5dff1e78282de95ce43116daebc83b765dbbf"}
Oct 30 20:44:01.453623 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:01.453468    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8efb31ead95e57a1035401f53ed5dff1e78282de95ce43116daebc83b765dbbf"
Oct 30 20:44:02.290430 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:02.290395    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"]
Oct 30 20:44:09.679091 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:44:09.679053    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:44:14.861744 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:14.861704    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"]
Oct 30 20:44:14.912524 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:14.912485    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c530ce-4gpqh"]
Oct 30 20:44:15.129995 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:15.129901    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:44:16.000858 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:16.000813    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" path="/var/lib/kubelet/pods/a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6/volumes"
Oct 30 20:44:17.310839 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.310785    4917 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n" podUID="660413ec-19a5-4698-8f14-e495c20addaf" containerName="main" containerID="cri-o://52d77552c08e124dc546b682d9d1a53630a4bb3bb9405f470794df1b9e2952b9" gracePeriod=45
Oct 30 20:44:17.534895 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.534864    4917 generic.go:356] "Generic (PLEG): container finished" podID="660413ec-19a5-4698-8f14-e495c20addaf" containerID="52d77552c08e124dc546b682d9d1a53630a4bb3bb9405f470794df1b9e2952b9" exitCode=0
Oct 30 20:44:17.535051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.534934    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n" event={"ID":"660413ec-19a5-4698-8f14-e495c20addaf","Type":"ContainerDied","Data":"52d77552c08e124dc546b682d9d1a53630a4bb3bb9405f470794df1b9e2952b9"}
Oct 30 20:44:17.579565 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.579538    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:44:17.660049 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660013    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf\" (UniqueName: \"kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf\") pod \"660413ec-19a5-4698-8f14-e495c20addaf\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") "
Oct 30 20:44:17.660217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660064    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache\") pod \"660413ec-19a5-4698-8f14-e495c20addaf\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") "
Oct 30 20:44:17.660217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660116    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir\") pod \"660413ec-19a5-4698-8f14-e495c20addaf\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") "
Oct 30 20:44:17.660217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660188    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs\") pod \"660413ec-19a5-4698-8f14-e495c20addaf\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") "
Oct 30 20:44:17.660217 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660216    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home\") pod \"660413ec-19a5-4698-8f14-e495c20addaf\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") "
Oct 30 20:44:17.660466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660245    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm\") pod \"660413ec-19a5-4698-8f14-e495c20addaf\" (UID: \"660413ec-19a5-4698-8f14-e495c20addaf\") "
Oct 30 20:44:17.660466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660384    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache" pod "660413ec-19a5-4698-8f14-e495c20addaf" (UID: "660413ec-19a5-4698-8f14-e495c20addaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:17.660466 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660435    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir" pod "660413ec-19a5-4698-8f14-e495c20addaf" (UID: "660413ec-19a5-4698-8f14-e495c20addaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:17.660628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660548    4917 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-tmp-dir\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:17.660628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660554    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home" pod "660413ec-19a5-4698-8f14-e495c20addaf" (UID: "660413ec-19a5-4698-8f14-e495c20addaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:17.660628 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.660568    4917 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-model-cache\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:17.662567 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.662540    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs" pod "660413ec-19a5-4698-8f14-e495c20addaf" (UID: "660413ec-19a5-4698-8f14-e495c20addaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:17.662667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.662567    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm" pod "660413ec-19a5-4698-8f14-e495c20addaf" (UID: "660413ec-19a5-4698-8f14-e495c20addaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:17.662857 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.662838    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf" pod "660413ec-19a5-4698-8f14-e495c20addaf" (UID: "660413ec-19a5-4698-8f14-e495c20addaf"). InnerVolumeSpecName "kube-api-access-c8wcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:17.761716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.761664    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8wcf\" (UniqueName: \"kubernetes.io/projected/660413ec-19a5-4698-8f14-e495c20addaf-kube-api-access-c8wcf\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:17.761716 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.761710    4917 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660413ec-19a5-4698-8f14-e495c20addaf-tls-certs\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:17.762027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.761729    4917 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-home\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:17.762027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:17.761746    4917 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660413ec-19a5-4698-8f14-e495c20addaf-dshm\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:18.542148 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:18.542114    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n" event={"ID":"660413ec-19a5-4698-8f14-e495c20addaf","Type":"ContainerDied","Data":"807b362c3c12abbdacbf94110e74ec1b6d4f026f7e6d995d5b0e2ed1a84c4108"}
Oct 30 20:44:18.542611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:18.542161    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"
Oct 30 20:44:18.542611 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:18.542163    4917 scope.go:122] "RemoveContainer" containerID="52d77552c08e124dc546b682d9d1a53630a4bb3bb9405f470794df1b9e2952b9"
Oct 30 20:44:18.709946 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:18.709910    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"]
Oct 30 20:44:18.760187 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:18.760156    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c530ce/simulator-single-main-c530ce-kserve-8676b4765-2dq4n"]
Oct 30 20:44:20.001085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:20.001047    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660413ec-19a5-4698-8f14-e495c20addaf" path="/var/lib/kubelet/pods/660413ec-19a5-4698-8f14-e495c20addaf/volumes"
Oct 30 20:44:34.775039 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:34.774998    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:44:39.679105 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:44:39.679067    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:44:41.063654 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:41.063618    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"]
Oct 30 20:44:41.113845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:41.113817    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942f676e-kmnxz"]
Oct 30 20:44:42.000861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:44:42.000824    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5378191b-ef74-49c6-9bcb-5ee41ec9147e" path="/var/lib/kubelet/pods/5378191b-ef74-49c6-9bcb-5ee41ec9147e/volumes"
Oct 30 20:45:09.678579 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:45:09.678542    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:45:14.994051 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:45:14.994015    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:45:39.678971 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:45:39.678887    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:45:42.794122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:45:42.794083    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:45:46.418189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:45:46.418140    4917 scope.go:122] "RemoveContainer" containerID="9633e3feaac92be3c9d0582a1a5122a7b849eaf3b05a0a536101ccc53688791d"
Oct 30 20:45:59.086185 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:45:59.086150    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:46:09.678796 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:46:09.678745    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:46:13.674193 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:13.674158    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:46:22.872868 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:22.872829    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:46:33.057737 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:33.057692    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:46:39.678592 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:46:39.678552    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:46:44.533597 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:44.533569    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:46:44.603968 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:44.603941    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pjlwh_9e23665b-8521-44d0-b403-ff1fefae7d74/console-operator/3.log"
Oct 30 20:46:46.441160 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:46.441130    4917 scope.go:122] "RemoveContainer" containerID="86d4602856ed9f00f9041163b2ed41065f57da5a66e750145b95be27ac756bea"
Oct 30 20:46:55.505667 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:46:55.505582    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:47:06.869140 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:06.869101    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:47:09.698594 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:47:09.698439    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:47:18.200528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:18.200490    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:47:26.642208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642171    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"]
Oct 30 20:47:26.642688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642566    4917 cpu_manager.go:414] "containerMap: removing container" podUID="a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" containerName="revoke-keys"
Oct 30 20:47:26.642688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642577    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" containerName="revoke-keys"
Oct 30 20:47:26.642688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642589    4917 cpu_manager.go:414] "containerMap: removing container" podUID="660413ec-19a5-4698-8f14-e495c20addaf" containerName="main"
Oct 30 20:47:26.642688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642594    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660413ec-19a5-4698-8f14-e495c20addaf" containerName="main"
Oct 30 20:47:26.642688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642660    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ba91f6-02ee-47d7-95fa-3dbc4b61eee6" containerName="revoke-keys"
Oct 30 20:47:26.642688 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.642670    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="660413ec-19a5-4698-8f14-e495c20addaf" containerName="main"
Oct 30 20:47:26.645661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.645644    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.688734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.688702    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.688734 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.688736    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdpj\" (UniqueName: \"kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.688982 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.688758    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.703819 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.703790    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"]
Oct 30 20:47:26.751873 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.751838    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:47:26.751873 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.751862    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:26.752089 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.751847    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:47:26.752089 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.751908    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:47:26.789656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.789624    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.789656 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.789658    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdpj\" (UniqueName: \"kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.789887 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.789680    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.790404 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.790381    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.792102 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.792077    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.852145 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.852111    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdpj\" (UniqueName: \"kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj\") pod \"maas-api-revoke-sub-f03629db-958c7\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:26.958456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:26.958425    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:27.330965 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:27.330940    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"]
Oct 30 20:47:27.332696 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:47:27.332668    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce8b5c_0e0d_4055_b59c_96996ab83f68.slice/crio-c40cb6fec2bf679c7204170c11523d140aa14bb45576e3cf29164f0ae0246266 WatchSource:0}: Error finding container c40cb6fec2bf679c7204170c11523d140aa14bb45576e3cf29164f0ae0246266: Status 404 returned error can't find the container with id c40cb6fec2bf679c7204170c11523d140aa14bb45576e3cf29164f0ae0246266
Oct 30 20:47:27.731650 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:27.731607    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7" event={"ID":"56ce8b5c-0e0d-4055-b59c-96996ab83f68","Type":"ContainerStarted","Data":"0dc7b90c2e24c801b5f75321de3e96a64ac24012a582f3e7dcb021b035e8cd78"}
Oct 30 20:47:27.732092 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:27.731669    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7" event={"ID":"56ce8b5c-0e0d-4055-b59c-96996ab83f68","Type":"ContainerStarted","Data":"c40cb6fec2bf679c7204170c11523d140aa14bb45576e3cf29164f0ae0246266"}
Oct 30 20:47:27.849002 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:27.848942    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7" podStartSLOduration=1.848927934 podStartE2EDuration="1.848927934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:27.848584388 +0000 UTC m=+3944.012986031" watchObservedRunningTime="2026-10-30 20:47:27.848927934 +0000 UTC m=+3944.013329603"
Oct 30 20:47:28.738718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:28.738686    4917 generic.go:356] "Generic (PLEG): container finished" podID="56ce8b5c-0e0d-4055-b59c-96996ab83f68" containerID="0dc7b90c2e24c801b5f75321de3e96a64ac24012a582f3e7dcb021b035e8cd78" exitCode=0
Oct 30 20:47:28.739143 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:28.738763    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7" event={"ID":"56ce8b5c-0e0d-4055-b59c-96996ab83f68","Type":"ContainerDied","Data":"0dc7b90c2e24c801b5f75321de3e96a64ac24012a582f3e7dcb021b035e8cd78"}
Oct 30 20:47:29.877395 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.877371    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:29.918015 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.917985    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca\") pod \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") "
Oct 30 20:47:29.918180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.918095    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca\") pod \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") "
Oct 30 20:47:29.918180 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.918141    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj\" (UniqueName: \"kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj\") pod \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\" (UID: \"56ce8b5c-0e0d-4055-b59c-96996ab83f68\") "
Oct 30 20:47:29.918435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.918411    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca" pod "56ce8b5c-0e0d-4055-b59c-96996ab83f68" (UID: "56ce8b5c-0e0d-4055-b59c-96996ab83f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:29.920219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.920199    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj" pod "56ce8b5c-0e0d-4055-b59c-96996ab83f68" (UID: "56ce8b5c-0e0d-4055-b59c-96996ab83f68"). InnerVolumeSpecName "kube-api-access-kwdpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:29.920312 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:29.920216    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca" pod "56ce8b5c-0e0d-4055-b59c-96996ab83f68" (UID: "56ce8b5c-0e0d-4055-b59c-96996ab83f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:30.019693 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:30.019661    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.019845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:30.019697    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdpj\" (UniqueName: \"kubernetes.io/projected/56ce8b5c-0e0d-4055-b59c-96996ab83f68-kube-api-access-kwdpj\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.019845 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:30.019712    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ce8b5c-0e0d-4055-b59c-96996ab83f68-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.753948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:30.753916    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7"
Oct 30 20:47:30.753948 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:30.753944    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-958c7" event={"ID":"56ce8b5c-0e0d-4055-b59c-96996ab83f68","Type":"ContainerDied","Data":"c40cb6fec2bf679c7204170c11523d140aa14bb45576e3cf29164f0ae0246266"}
Oct 30 20:47:30.754168 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:30.753984    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c40cb6fec2bf679c7204170c11523d140aa14bb45576e3cf29164f0ae0246266"
Oct 30 20:47:39.361085 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:39.361044    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:47:39.679092 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:47:39.679053    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:47:47.039050 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.039016    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"]
Oct 30 20:47:47.039456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.039417    4917 cpu_manager.go:414] "containerMap: removing container" podUID="56ce8b5c-0e0d-4055-b59c-96996ab83f68" containerName="revoke-keys"
Oct 30 20:47:47.039456 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.039430    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ce8b5c-0e0d-4055-b59c-96996ab83f68" containerName="revoke-keys"
Oct 30 20:47:47.039534 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.039512    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ce8b5c-0e0d-4055-b59c-96996ab83f68" containerName="revoke-keys"
Oct 30 20:47:47.042937 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.042912    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.097861 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.097824    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:47.100813 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.100785    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"]
Oct 30 20:47:47.148026 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.147995    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:47:47.148199 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.148054    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:47:47.148294 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.148228    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:47:47.210704 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.210676    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.210874 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.210720    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.210922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.210894    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wx8z\" (UniqueName: \"kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.312031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.311958    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.312031 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.312022    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx8z\" (UniqueName: \"kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.312208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.312073    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.312745 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.312727    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.314385 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.314368    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.374350 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.374319    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wx8z\" (UniqueName: \"kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z\") pod \"maas-api-revoke-sub-0165d7d2-9kzv4\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:47.655090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:47.654999    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:48.028033 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:48.028003    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"]
Oct 30 20:47:48.030291 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:47:48.030264    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd012cd4_3145_46f1_844d_a522c9ae88f9.slice/crio-1933cae2e46f61ff04834a3eeb53d9f4fe69a8f20a226d9dece257d5c8c4a6c1 WatchSource:0}: Error finding container 1933cae2e46f61ff04834a3eeb53d9f4fe69a8f20a226d9dece257d5c8c4a6c1: Status 404 returned error can't find the container with id 1933cae2e46f61ff04834a3eeb53d9f4fe69a8f20a226d9dece257d5c8c4a6c1
Oct 30 20:47:48.849163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:48.849124    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4" event={"ID":"fd012cd4-3145-46f1-844d-a522c9ae88f9","Type":"ContainerStarted","Data":"3352bebf2e428fa3209590071755cea3a7bb04c22b92d4b9ad56c20b342410af"}
Oct 30 20:47:48.849163 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:48.849180    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4" event={"ID":"fd012cd4-3145-46f1-844d-a522c9ae88f9","Type":"ContainerStarted","Data":"1933cae2e46f61ff04834a3eeb53d9f4fe69a8f20a226d9dece257d5c8c4a6c1"}
Oct 30 20:47:48.967544 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:48.967497    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4" podStartSLOduration=2.967484215 podStartE2EDuration="2.967484215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:48.965647667 +0000 UTC m=+3965.130049319" watchObservedRunningTime="2026-10-30 20:47:48.967484215 +0000 UTC m=+3965.131885852"
Oct 30 20:47:49.855402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:49.855369    4917 generic.go:356] "Generic (PLEG): container finished" podID="fd012cd4-3145-46f1-844d-a522c9ae88f9" containerID="3352bebf2e428fa3209590071755cea3a7bb04c22b92d4b9ad56c20b342410af" exitCode=0
Oct 30 20:47:49.855826 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:49.855461    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4" event={"ID":"fd012cd4-3145-46f1-844d-a522c9ae88f9","Type":"ContainerDied","Data":"3352bebf2e428fa3209590071755cea3a7bb04c22b92d4b9ad56c20b342410af"}
Oct 30 20:47:50.996117 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:50.996049    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:51.148090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.148016    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z\" (UniqueName: \"kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z\") pod \"fd012cd4-3145-46f1-844d-a522c9ae88f9\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") "
Oct 30 20:47:51.148090 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.148068    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca\") pod \"fd012cd4-3145-46f1-844d-a522c9ae88f9\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") "
Oct 30 20:47:51.148280 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.148160    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca\") pod \"fd012cd4-3145-46f1-844d-a522c9ae88f9\" (UID: \"fd012cd4-3145-46f1-844d-a522c9ae88f9\") "
Oct 30 20:47:51.148528 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.148505    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca" pod "fd012cd4-3145-46f1-844d-a522c9ae88f9" (UID: "fd012cd4-3145-46f1-844d-a522c9ae88f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:51.150215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.150191    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca" pod "fd012cd4-3145-46f1-844d-a522c9ae88f9" (UID: "fd012cd4-3145-46f1-844d-a522c9ae88f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:51.150322 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.150276    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z" pod "fd012cd4-3145-46f1-844d-a522c9ae88f9" (UID: "fd012cd4-3145-46f1-844d-a522c9ae88f9"). InnerVolumeSpecName "kube-api-access-6wx8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:51.249783 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.249747    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:51.249932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.249799    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wx8z\" (UniqueName: \"kubernetes.io/projected/fd012cd4-3145-46f1-844d-a522c9ae88f9-kube-api-access-6wx8z\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:51.249932 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.249810    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd012cd4-3145-46f1-844d-a522c9ae88f9-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:51.867485 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.867448    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4"
Oct 30 20:47:51.867662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.867481    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9kzv4" event={"ID":"fd012cd4-3145-46f1-844d-a522c9ae88f9","Type":"ContainerDied","Data":"1933cae2e46f61ff04834a3eeb53d9f4fe69a8f20a226d9dece257d5c8c4a6c1"}
Oct 30 20:47:51.867662 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:51.867532    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1933cae2e46f61ff04834a3eeb53d9f4fe69a8f20a226d9dece257d5c8c4a6c1"
Oct 30 20:47:58.361592 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.361549    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"]
Oct 30 20:47:58.362252 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.362230    4917 cpu_manager.go:414] "containerMap: removing container" podUID="fd012cd4-3145-46f1-844d-a522c9ae88f9" containerName="revoke-keys"
Oct 30 20:47:58.362307 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.362254    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd012cd4-3145-46f1-844d-a522c9ae88f9" containerName="revoke-keys"
Oct 30 20:47:58.362375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.362362    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd012cd4-3145-46f1-844d-a522c9ae88f9" containerName="revoke-keys"
Oct 30 20:47:58.366014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.365989    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.408027 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.407992    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.408215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.408031    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d962\" (UniqueName: \"kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.408215 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.408150    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.421929 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.421899    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"]
Oct 30 20:47:58.469365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.469328    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:47:58.469365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.469349    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:47:58.469365 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.469336    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:58.469683 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.469667    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:47:58.509362 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.509330    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.509533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.509434    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.509533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.509473    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d962\" (UniqueName: \"kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.510131 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.510110    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.511838 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.511818    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.571435 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.571402    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d962\" (UniqueName: \"kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962\") pod \"maas-api-revoke-sub-6b19338f-p77kw\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:58.679003 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:58.678973    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:47:59.051852 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:59.051826    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"]
Oct 30 20:47:59.053359 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:47:59.053329    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76013cde_b714_4c12_a2b3_b8235d0f7d69.slice/crio-cc1c7240b66bdbf1abb6c05b5dc341c75a1b69002a3835b505c2fef15231d484 WatchSource:0}: Error finding container cc1c7240b66bdbf1abb6c05b5dc341c75a1b69002a3835b505c2fef15231d484: Status 404 returned error can't find the container with id cc1c7240b66bdbf1abb6c05b5dc341c75a1b69002a3835b505c2fef15231d484
Oct 30 20:47:59.910081 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:59.910042    4917 generic.go:356] "Generic (PLEG): container finished" podID="76013cde-b714-4c12-a2b3-b8235d0f7d69" containerID="61a0338595e2f77f408f2e13bca619d30061e468c8a9b54efa5461232cbe5fec" exitCode=0
Oct 30 20:47:59.910539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:59.910092    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw" event={"ID":"76013cde-b714-4c12-a2b3-b8235d0f7d69","Type":"ContainerDied","Data":"61a0338595e2f77f408f2e13bca619d30061e468c8a9b54efa5461232cbe5fec"}
Oct 30 20:47:59.910539 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:47:59.910152    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw" event={"ID":"76013cde-b714-4c12-a2b3-b8235d0f7d69","Type":"ContainerStarted","Data":"cc1c7240b66bdbf1abb6c05b5dc341c75a1b69002a3835b505c2fef15231d484"}
Oct 30 20:48:01.055096 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.055071    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:48:01.130922 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.130886    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca\") pod \"76013cde-b714-4c12-a2b3-b8235d0f7d69\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") "
Oct 30 20:48:01.131094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.130944    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962\" (UniqueName: \"kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962\") pod \"76013cde-b714-4c12-a2b3-b8235d0f7d69\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") "
Oct 30 20:48:01.131094 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.130993    4917 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca\") pod \"76013cde-b714-4c12-a2b3-b8235d0f7d69\" (UID: \"76013cde-b714-4c12-a2b3-b8235d0f7d69\") "
Oct 30 20:48:01.131412 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.131381    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca" pod "76013cde-b714-4c12-a2b3-b8235d0f7d69" (UID: "76013cde-b714-4c12-a2b3-b8235d0f7d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:01.133210 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.133183    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca" pod "76013cde-b714-4c12-a2b3-b8235d0f7d69" (UID: "76013cde-b714-4c12-a2b3-b8235d0f7d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:01.133313 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.133255    4917 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962" pod "76013cde-b714-4c12-a2b3-b8235d0f7d69" (UID: "76013cde-b714-4c12-a2b3-b8235d0f7d69"). InnerVolumeSpecName "kube-api-access-2d962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:01.232414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.232381    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-xks-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:01.232414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.232412    4917 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d962\" (UniqueName: \"kubernetes.io/projected/76013cde-b714-4c12-a2b3-b8235d0f7d69-kube-api-access-2d962\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:01.232638 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.232425    4917 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76013cde-b714-4c12-a2b3-b8235d0f7d69-maas-api-openshift-ca\") on node \"ip-10-0-133-27.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:01.922666 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.922627    4917 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw"
Oct 30 20:48:01.922914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.922627    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p77kw" event={"ID":"76013cde-b714-4c12-a2b3-b8235d0f7d69","Type":"ContainerDied","Data":"cc1c7240b66bdbf1abb6c05b5dc341c75a1b69002a3835b505c2fef15231d484"}
Oct 30 20:48:01.922914 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:01.922765    4917 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc1c7240b66bdbf1abb6c05b5dc341c75a1b69002a3835b505c2fef15231d484"
Oct 30 20:48:09.679252 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:48:09.679212    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:48:11.064701 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:11.064666    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkr62"]
Oct 30 20:48:22.526446 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:22.526408    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xht5c_52325904-1a6d-4d6b-9766-132c039f063f/manager/0.log"
Oct 30 20:48:22.814348 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:22.814259    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-5rp7h_625ee131-9ed8-4e2e-a591-66298aed7e86/manager/0.log"
Oct 30 20:48:23.144219 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:23.144142    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5pdj6_eec900b3-e382-413e-aa96-c42ca854041d/manager/0.log"
Oct 30 20:48:24.153712 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:24.153629    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wwt22_c297f662-710f-4a22-af65-ca401969d921/server/0.log"
Oct 30 20:48:25.762442 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:25.762408    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps_29cef479-2bf6-49a6-b680-4da0a7c44b56/copy-manifests/0.log"
Oct 30 20:48:25.832903 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:25.832872    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps_29cef479-2bf6-49a6-b680-4da0a7c44b56/manager/0.log"
Oct 30 20:48:27.628663 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:27.628631    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-9kzv4_fd012cd4-3145-46f1-844d-a522c9ae88f9/revoke-keys/0.log"
Oct 30 20:48:28.562902 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:28.562878    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-p77kw_76013cde-b714-4c12-a2b3-b8235d0f7d69/revoke-keys/0.log"
Oct 30 20:48:28.801082 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:28.801041    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-757f63d0-x2ckq_a70edaeb-81c7-4079-b319-e4a3008c444a/revoke-keys/0.log"
Oct 30 20:48:29.045156 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:29.045126    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-958c7_56ce8b5c-0e0d-4055-b59c-96996ab83f68/revoke-keys/0.log"
Oct 30 20:48:31.992125 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:31.992101    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sddkh_e6809a5a-77c6-46e1-9738-97f6d969b2ac/kuadrant-console-plugin/0.log"
Oct 30 20:48:32.812349 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:32.812321    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkr62_9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56/limitador/0.log"
Oct 30 20:48:33.893431 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:33.893392    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-h9wmt_a75eeff5-1531-4c77-9c57-b2bcd81d5871/istio-proxy/0.log"
Oct 30 20:48:34.930130 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:34.930087    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-2zdhm_787192ab-bdfa-4cbc-9b85-bc2eaad7f101/istio-proxy/0.log"
Oct 30 20:48:36.300634 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:36.300604    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-fc6zk_e680d509-ed58-4150-bcdb-608c9dc0f244/istio-proxy/0.log"
Oct 30 20:48:36.861266 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:36.861232    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-kt69c_b17447df-4223-406e-8271-9c6d9b5e15a6/istio-proxy/0.log"
Oct 30 20:48:38.133242 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:38.133212    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569cc6f5cf-pw4pl_5b117693-4843-4840-a6f1-5567a04226e1/router/0.log"
Oct 30 20:48:39.678084 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:48:39.678044    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:48:50.056330 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:50.056290    4917 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"]
Oct 30 20:48:50.106846 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:50.106803    4917 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-757f63d0-x2ckq"]
Oct 30 20:48:51.737261 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:51.737229    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xht5c_52325904-1a6d-4d6b-9766-132c039f063f/manager/0.log"
Oct 30 20:48:52.001702 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:52.001621    4917 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70edaeb-81c7-4079-b319-e4a3008c444a" path="/var/lib/kubelet/pods/a70edaeb-81c7-4079-b319-e4a3008c444a/volumes"
Oct 30 20:48:52.030807 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:52.030754    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-5rp7h_625ee131-9ed8-4e2e-a591-66298aed7e86/manager/0.log"
Oct 30 20:48:52.383596 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:52.383505    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5pdj6_eec900b3-e382-413e-aa96-c42ca854041d/manager/0.log"
Oct 30 20:48:53.108014 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:53.107988    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wwt22_c297f662-710f-4a22-af65-ca401969d921/server/0.log"
Oct 30 20:48:54.841931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:54.841904    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps_29cef479-2bf6-49a6-b680-4da0a7c44b56/copy-manifests/0.log"
Oct 30 20:48:54.913441 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:54.913410    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-fd6ps_29cef479-2bf6-49a6-b680-4da0a7c44b56/manager/0.log"
Oct 30 20:48:56.550522 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:56.550491    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-9kzv4_fd012cd4-3145-46f1-844d-a522c9ae88f9/revoke-keys/0.log"
Oct 30 20:48:57.459991 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:57.459960    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-p77kw_76013cde-b714-4c12-a2b3-b8235d0f7d69/revoke-keys/0.log"
Oct 30 20:48:57.693661 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:48:57.693630    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-958c7_56ce8b5c-0e0d-4055-b59c-96996ab83f68/revoke-keys/0.log"
Oct 30 20:49:00.067257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:00.067232    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sddkh_e6809a5a-77c6-46e1-9738-97f6d969b2ac/kuadrant-console-plugin/0.log"
Oct 30 20:49:00.837354 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:00.837313    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkr62_9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56/limitador/0.log"
Oct 30 20:49:01.717122 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:01.717088    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-h9wmt_a75eeff5-1531-4c77-9c57-b2bcd81d5871/istio-proxy/0.log"
Oct 30 20:49:02.739393 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:02.739359    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-2zdhm_787192ab-bdfa-4cbc-9b85-bc2eaad7f101/istio-proxy/0.log"
Oct 30 20:49:04.144498 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:04.144459    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-fc6zk_e680d509-ed58-4150-bcdb-608c9dc0f244/istio-proxy/0.log"
Oct 30 20:49:04.718269 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:04.718239    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-kt69c_b17447df-4223-406e-8271-9c6d9b5e15a6/istio-proxy/0.log"
Oct 30 20:49:06.083397 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:49:06.083368    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569cc6f5cf-pw4pl_5b117693-4843-4840-a6f1-5567a04226e1/router/0.log"
Oct 30 20:49:09.679248 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:49:09.679205    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:49:39.679515 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:49:39.679464    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:50:09.678928 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:50:09.678890    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:50:18.808228 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.808186    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t2ndh/must-gather-sw92h"]
Oct 30 20:50:18.808739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.808592    4917 cpu_manager.go:414] "containerMap: removing container" podUID="76013cde-b714-4c12-a2b3-b8235d0f7d69" containerName="revoke-keys"
Oct 30 20:50:18.808739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.808603    4917 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76013cde-b714-4c12-a2b3-b8235d0f7d69" containerName="revoke-keys"
Oct 30 20:50:18.808739 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.808679    4917 memory_manager.go:365] "RemoveStaleState removing state" podUID="76013cde-b714-4c12-a2b3-b8235d0f7d69" containerName="revoke-keys"
Oct 30 20:50:18.811870 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.811853    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:18.864901 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.864873    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t2ndh\"/\"openshift-service-ca.crt\""
Oct 30 20:50:18.865189 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.865167    4917 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t2ndh\"/\"kube-root-ca.crt\""
Oct 30 20:50:18.873668 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.873647    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31afeb94-5948-4f42-b055-a4c1b825456f-must-gather-output\") pod \"must-gather-sw92h\" (UID: \"31afeb94-5948-4f42-b055-a4c1b825456f\") " pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:18.873740 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.873710    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8r9\" (UniqueName: \"kubernetes.io/projected/31afeb94-5948-4f42-b055-a4c1b825456f-kube-api-access-qx8r9\") pod \"must-gather-sw92h\" (UID: \"31afeb94-5948-4f42-b055-a4c1b825456f\") " pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:18.915351 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.915321    4917 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t2ndh\"/\"default-dockercfg-zhp6b\""
Oct 30 20:50:18.917899 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.917877    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/must-gather-sw92h"]
Oct 30 20:50:18.974942 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.974911    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8r9\" (UniqueName: \"kubernetes.io/projected/31afeb94-5948-4f42-b055-a4c1b825456f-kube-api-access-qx8r9\") pod \"must-gather-sw92h\" (UID: \"31afeb94-5948-4f42-b055-a4c1b825456f\") " pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:18.975123 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.975039    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31afeb94-5948-4f42-b055-a4c1b825456f-must-gather-output\") pod \"must-gather-sw92h\" (UID: \"31afeb94-5948-4f42-b055-a4c1b825456f\") " pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:18.975402 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:18.975383    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31afeb94-5948-4f42-b055-a4c1b825456f-must-gather-output\") pod \"must-gather-sw92h\" (UID: \"31afeb94-5948-4f42-b055-a4c1b825456f\") " pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:19.036677 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:19.036645    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8r9\" (UniqueName: \"kubernetes.io/projected/31afeb94-5948-4f42-b055-a4c1b825456f-kube-api-access-qx8r9\") pod \"must-gather-sw92h\" (UID: \"31afeb94-5948-4f42-b055-a4c1b825456f\") " pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:19.126536 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:19.126461    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/must-gather-sw92h"
Oct 30 20:50:19.505794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:19.505740    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/must-gather-sw92h"]
Oct 30 20:50:19.508256 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:50:19.508225    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31afeb94_5948_4f42_b055_a4c1b825456f.slice/crio-dff805477834fe57618c5604a4f02c12774e702659dc2e1df70bfa60f186178d WatchSource:0}: Error finding container dff805477834fe57618c5604a4f02c12774e702659dc2e1df70bfa60f186178d: Status 404 returned error can't find the container with id dff805477834fe57618c5604a4f02c12774e702659dc2e1df70bfa60f186178d
Oct 30 20:50:19.510482 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:19.510465    4917 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:50:19.645172 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:19.645138    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/must-gather-sw92h" event={"ID":"31afeb94-5948-4f42-b055-a4c1b825456f","Type":"ContainerStarted","Data":"dff805477834fe57618c5604a4f02c12774e702659dc2e1df70bfa60f186178d"}
Oct 30 20:50:21.659076 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:21.659026    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/must-gather-sw92h" event={"ID":"31afeb94-5948-4f42-b055-a4c1b825456f","Type":"ContainerStarted","Data":"f6de19b9049d946872393f1ea92f36f3dbae785762d20b69c6bee474a57dc789"}
Oct 30 20:50:21.659533 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:21.659085    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/must-gather-sw92h" event={"ID":"31afeb94-5948-4f42-b055-a4c1b825456f","Type":"ContainerStarted","Data":"4d99e33595182a9499250b5d978e8aab96e429f30005ff1f4463ade837496214"}
Oct 30 20:50:21.777527 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:21.777147    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t2ndh/must-gather-sw92h" podStartSLOduration=2.499031987 podStartE2EDuration="3.777128294s" totalImagesPullingTime="1.278096307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:21.775475512 +0000 UTC m=+4117.939877155" watchObservedRunningTime="2026-10-30 20:50:21.777128294 +0000 UTC m=+4117.941529937"
Oct 30 20:50:29.123555 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:29.123521    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x6kbk_710b36be-68a9-4ed5-84c6-30d5e3dc6801/global-pull-secret-syncer/0.log"
Oct 30 20:50:29.731375 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:29.731342    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8wclq_19670c03-8928-4c59-b153-b51d3a77f63e/connection-checker/0.log"
Oct 30 20:50:30.162414 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:30.162338    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-swx4r_2cc8f2c8-4792-4fbf-b958-655692c689c7/connection-checker/0.log"
Oct 30 20:50:30.610803 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:30.607194    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-txjbg_7ae66fed-db02-4f76-99ec-a80127e2a76a/connection-checker/0.log"
Oct 30 20:50:32.211022 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:32.210975    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tznqp_4f1128e1-21b4-437a-a7f5-e94567bec795/konnectivity-agent/0.log"
Oct 30 20:50:33.036795 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:33.036553    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-27.us-west-1.compute.internal_5d870f2d14f468b90253c9dd98ce2b16/haproxy/0.log"
Oct 30 20:50:37.775802 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.774509    4917 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"]
Oct 30 20:50:37.779957 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.779896    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.857208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.857175    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-sys\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.857208 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.857214    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-proc\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.857425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.857244    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldvb\" (UniqueName: \"kubernetes.io/projected/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-kube-api-access-bldvb\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.857425 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.857392    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-podres\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.857503 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.857425    4917 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-lib-modules\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.868129 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.868090    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"]
Oct 30 20:50:37.959251 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959218    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-sys\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959260    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-proc\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959365    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldvb\" (UniqueName: \"kubernetes.io/projected/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-kube-api-access-bldvb\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959366    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-sys\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959438 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959431    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-podres\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959440    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-proc\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959454    4917 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-lib-modules\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959499    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-podres\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:37.959642 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:37.959531    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-lib-modules\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:38.020521 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.020487    4917 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldvb\" (UniqueName: \"kubernetes.io/projected/f1afc108-1fc9-42ca-be4a-f2b66b457d3b-kube-api-access-bldvb\") pod \"perf-node-gather-daemonset-k98nz\" (UID: \"f1afc108-1fc9-42ca-be4a-f2b66b457d3b\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:38.097718 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.097634    4917 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:38.292064 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.292037    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sddkh_e6809a5a-77c6-46e1-9738-97f6d969b2ac/kuadrant-console-plugin/0.log"
Oct 30 20:50:38.473649 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.473625    4917 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"]
Oct 30 20:50:38.475346 ip-10-0-133-27 kubenswrapper[4917]: W1030 20:50:38.475313    4917 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1afc108_1fc9_42ca_be4a_f2b66b457d3b.slice/crio-04284ae320679c07eb845ade6459084068de9d1efac74292a34aa6650c1a3cc0 WatchSource:0}: Error finding container 04284ae320679c07eb845ade6459084068de9d1efac74292a34aa6650c1a3cc0: Status 404 returned error can't find the container with id 04284ae320679c07eb845ade6459084068de9d1efac74292a34aa6650c1a3cc0
Oct 30 20:50:38.823067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.822970    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz" event={"ID":"f1afc108-1fc9-42ca-be4a-f2b66b457d3b","Type":"ContainerStarted","Data":"6a3e56f766ddbe83d33629a069fce34a5ec241a5892da3db41118a322235557d"}
Oct 30 20:50:38.823067 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.823043    4917 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz" event={"ID":"f1afc108-1fc9-42ca-be4a-f2b66b457d3b","Type":"ContainerStarted","Data":"04284ae320679c07eb845ade6459084068de9d1efac74292a34aa6650c1a3cc0"}
Oct 30 20:50:38.823794 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.823753    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:38.941636 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:38.941578    4917 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz" podStartSLOduration=1.941565775 podStartE2EDuration="1.941565775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:38.939864892 +0000 UTC m=+4135.104266533" watchObservedRunningTime="2026-10-30 20:50:38.941565775 +0000 UTC m=+4135.105967414"
Oct 30 20:50:39.383298 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:39.383266    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkr62_9d9b5e5e-f34e-4513-98f0-a6f79a1f3f56/limitador/0.log"
Oct 30 20:50:39.681839 ip-10-0-133-27 kubenswrapper[4917]: E1030 20:50:39.681799    4917 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nwlvj" containerName="extractor"
Oct 30 20:50:45.472190 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:45.472161    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kk9wf_bc01cb5a-688e-4b24-ab7b-8157db6ccc1f/cluster-monitoring-operator/0.log"
Oct 30 20:50:45.845255 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:45.845183    4917 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-k98nz"
Oct 30 20:50:46.508221 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:46.508188    4917 scope.go:122] "RemoveContainer" containerID="0cd06ea0431ae190f4ecb7819712813c5186fd3c670d43aa6a3fbb53740b6185"
Oct 30 20:50:46.518951 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:46.518922    4917 scope.go:122] "RemoveContainer" containerID="8472b265f6bd8a70ccde0a7bf5ae1da8be9267461bff4180439713917a678472"
Oct 30 20:50:49.385241 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:49.385207    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lmdpj_a0b293ce-2016-4abb-ae21-9e88058f087f/node-exporter/0.log"
Oct 30 20:50:49.803931 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:49.803887    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lmdpj_a0b293ce-2016-4abb-ae21-9e88058f087f/kube-rbac-proxy/0.log"
Oct 30 20:50:50.224043 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:50:50.224019    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lmdpj_a0b293ce-2016-4abb-ae21-9e88058f087f/init-textfile/0.log"
Oct 30 20:51:00.900257 ip-10-0-133-27 kubenswrapper[4917]: I1030 20:51:00.900224    4917 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-h9wmt_a75eeff5-1531-4c77-9c57-b2bcd81d5871/istio-proxy/0.log"